Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 07:58:08.630739 18486 test_util.cc:268] Using random seed: -155278400
I20240502 07:58:08.637394 18486 ts_itest-base.cc:112] Starting cluster with:
I20240502 07:58:08.637501 18486 ts_itest-base.cc:113] --------------
I20240502 07:58:08.637534 18486 ts_itest-base.cc:114] 6 tablet servers
I20240502 07:58:08.637557 18486 ts_itest-base.cc:115] 3 replicas per TS
I20240502 07:58:08.637585 18486 ts_itest-base.cc:116] --------------
2024-05-02T07:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T07:58:08Z Disabled control of system clock
I20240502 07:58:08.646955 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.13.190:46263
--webserver_interface=127.18.13.190
--webserver_port=0
--builtin_ntp_servers=127.18.13.148:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.13.190:46263
--location_mapping_cmd=/tmp/dist-test-taskeLuhys/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 07:58:08.757236 18498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:08.757452 18498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:08.757495 18498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:08.760520 18498 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:08.760584 18498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 07:58:08.760608 18498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:08.760646 18498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 07:58:08.760686 18498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 07:58:08.764279 18498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskeLuhys/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.18.13.190:46263
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.13.190:46263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.18.13.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.18498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:08.765079 18498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:08.766166 18498 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:08.774566 18504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:08.775367 18503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:08.776546 18506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:08.777716 18498 server_base.cc:1034] running on GCE node
I20240502 07:58:08.778165 18498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:08.779004 18498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:08.780151 18498 hybrid_clock.cc:648] HybridClock initialized: now 1714636688780149 us; error 33 us; skew 500 ppm
I20240502 07:58:08.787273 18498 webserver.cc:460] Webserver started at http://127.18.13.190:44101/ using document root <none> and password file <none>
I20240502 07:58:08.787868 18498 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:08.787928 18498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:08.788127 18498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:08.789763 18498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7375e73262114a86b5bdd384b5c786ee"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-f46z"
I20240502 07:58:08.790215 18498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7375e73262114a86b5bdd384b5c786ee"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-f46z"
I20240502 07:58:08.793398 18498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:58:08.795488 18512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:08.796489 18498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:08.796607 18498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7375e73262114a86b5bdd384b5c786ee"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-f46z"
I20240502 07:58:08.796686 18498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:08.811681 18498 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:08.812273 18498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:08.812422 18498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:08.818851 18498 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.190:46263
I20240502 07:58:08.818864 18564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.190:46263 every 8 connection(s)
I20240502 07:58:08.819823 18498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 07:58:08.822285 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 18498
I20240502 07:58:08.822428 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 07:58:08.822444 18565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:08.828464 18565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee: Bootstrap starting.
I20240502 07:58:08.830715 18565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:08.831526 18565 log.cc:824] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:08.833225 18565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee: No bootstrap required, opened a new log
I20240502 07:58:08.836289 18565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7375e73262114a86b5bdd384b5c786ee" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46263 } }
I20240502 07:58:08.836539 18565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:08.836618 18565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7375e73262114a86b5bdd384b5c786ee, State: Initialized, Role: FOLLOWER
I20240502 07:58:08.837046 18565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7375e73262114a86b5bdd384b5c786ee" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46263 } }
I20240502 07:58:08.837170 18565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:08.837242 18565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:08.837349 18565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:08.838228 18565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7375e73262114a86b5bdd384b5c786ee" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46263 } }
I20240502 07:58:08.838572 18565 leader_election.cc:302] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7375e73262114a86b5bdd384b5c786ee; no voters:
I20240502 07:58:08.838864 18565 leader_election.cc:288] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:58:08.839314 18570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:08.839507 18570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [term 1 LEADER]: Becoming Leader. State: Replica: 7375e73262114a86b5bdd384b5c786ee, State: Running, Role: LEADER
I20240502 07:58:08.839814 18565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:58:08.839771 18570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7375e73262114a86b5bdd384b5c786ee" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46263 } }
I20240502 07:58:08.841233 18572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7375e73262114a86b5bdd384b5c786ee. Latest consensus state: current_term: 1 leader_uuid: "7375e73262114a86b5bdd384b5c786ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7375e73262114a86b5bdd384b5c786ee" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46263 } } }
I20240502 07:58:08.841250 18571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7375e73262114a86b5bdd384b5c786ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7375e73262114a86b5bdd384b5c786ee" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46263 } } }
I20240502 07:58:08.841354 18572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:08.841374 18571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:08.841681 18579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:58:08.844509 18579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:58:08.849975 18579 catalog_manager.cc:1347] Generated new cluster ID: 73effd8779fd42759c2c4511a2d1a003
I20240502 07:58:08.850134 18579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:58:08.867064 18579 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:58:08.868314 18579 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:58:08.874368 18579 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7375e73262114a86b5bdd384b5c786ee: Generated new TSK 0
I20240502 07:58:08.875130 18579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:58:08.880162 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.129:0
--local_ip_for_outbound_sockets=127.18.13.129
--webserver_interface=127.18.13.129
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46263
--builtin_ntp_servers=127.18.13.148:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 07:58:08.984814 18589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:08.985035 18589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:08.985117 18589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:08.988135 18589 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:08.988211 18589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:08.988307 18589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.129
I20240502 07:58:08.991879 18589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.18.13.129
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46263
--never_fsync=true
--heap_profile_path=/tmp/kudu.18589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.129
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:08.992758 18589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:08.993870 18589 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:09.002981 18597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.004099 18595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.005045 18594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.005908 18589 server_base.cc:1034] running on GCE node
I20240502 07:58:09.006299 18589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:09.006873 18589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:09.008031 18589 hybrid_clock.cc:648] HybridClock initialized: now 1714636689008007 us; error 38 us; skew 500 ppm
I20240502 07:58:09.009869 18589 webserver.cc:460] Webserver started at http://127.18.13.129:45109/ using document root <none> and password file <none>
I20240502 07:58:09.010429 18589 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.010491 18589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.010660 18589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.012354 18589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2f6bc76c600c413d9f72526563d3234a"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.012816 18589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2f6bc76c600c413d9f72526563d3234a"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.016120 18589 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 07:58:09.018330 18603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.019366 18589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:09.019470 18589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2f6bc76c600c413d9f72526563d3234a"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.019554 18589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:09.040509 18589 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:09.041175 18589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:09.041330 18589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.041857 18589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:09.042711 18589 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:09.042757 18589 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.042794 18589 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:09.042812 18589 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.051203 18589 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.129:40523
I20240502 07:58:09.051236 18716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.129:40523 every 8 connection(s)
I20240502 07:58:09.052153 18589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 07:58:09.056116 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 18589
I20240502 07:58:09.056202 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 07:58:09.058607 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.130:0
--local_ip_for_outbound_sockets=127.18.13.130
--webserver_interface=127.18.13.130
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46263
--builtin_ntp_servers=127.18.13.148:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:09.061952 18717 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46263
I20240502 07:58:09.062227 18717 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.062919 18717 heartbeater.cc:511] Master 127.18.13.190:46263 requested a full tablet report, sending...
I20240502 07:58:09.103201 18529 ts_manager.cc:194] Registered new tserver with Master: 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:09.104672 18529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.129:57597
W20240502 07:58:09.169286 18720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:09.169509 18720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:09.169574 18720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:09.172612 18720 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:09.172683 18720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:09.172770 18720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.130
I20240502 07:58:09.176385 18720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.18.13.130
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46263
--never_fsync=true
--heap_profile_path=/tmp/kudu.18720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.130
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:09.177294 18720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:09.178378 18720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:09.187076 18726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.188164 18729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.189122 18727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.190250 18720 server_base.cc:1034] running on GCE node
I20240502 07:58:09.190667 18720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:09.191211 18720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:09.192358 18720 hybrid_clock.cc:648] HybridClock initialized: now 1714636689192348 us; error 30 us; skew 500 ppm
I20240502 07:58:09.194202 18720 webserver.cc:460] Webserver started at http://127.18.13.130:37219/ using document root <none> and password file <none>
I20240502 07:58:09.194748 18720 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.194809 18720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.195057 18720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.196717 18720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4135e6d90aec4830b08bcb7e8dc50260"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.197186 18720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4135e6d90aec4830b08bcb7e8dc50260"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.200436 18720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 07:58:09.202524 18735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.203497 18720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:09.203608 18720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4135e6d90aec4830b08bcb7e8dc50260"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.203691 18720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:09.225821 18720 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:09.226477 18720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:09.226646 18720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.227206 18720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:09.228173 18720 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:09.228242 18720 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.228302 18720 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:09.228379 18720 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.237902 18720 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.130:43999
I20240502 07:58:09.237927 18848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.130:43999 every 8 connection(s)
I20240502 07:58:09.238876 18720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 07:58:09.244951 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 18720
I20240502 07:58:09.245114 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 07:58:09.247547 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.131:0
--local_ip_for_outbound_sockets=127.18.13.131
--webserver_interface=127.18.13.131
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46263
--builtin_ntp_servers=127.18.13.148:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:09.248517 18849 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46263
I20240502 07:58:09.248756 18849 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.249308 18849 heartbeater.cc:511] Master 127.18.13.190:46263 requested a full tablet report, sending...
I20240502 07:58:09.306195 18529 ts_manager.cc:194] Registered new tserver with Master: 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999)
I20240502 07:58:09.307135 18529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.130:51545
W20240502 07:58:09.376564 18852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:09.376830 18852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:09.376900 18852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:09.379984 18852 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:09.380064 18852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:09.380162 18852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.131
I20240502 07:58:09.384001 18852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.18.13.131
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46263
--never_fsync=true
--heap_profile_path=/tmp/kudu.18852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.131
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:09.384917 18852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:09.386046 18852 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:09.395215 18858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.396340 18859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.397392 18852 server_base.cc:1034] running on GCE node
W20240502 07:58:09.398265 18861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.398691 18852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:09.399308 18852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:09.400504 18852 hybrid_clock.cc:648] HybridClock initialized: now 1714636689400488 us; error 37 us; skew 500 ppm
I20240502 07:58:09.402679 18852 webserver.cc:460] Webserver started at http://127.18.13.131:43107/ using document root <none> and password file <none>
I20240502 07:58:09.403271 18852 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.403329 18852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.403513 18852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.405316 18852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9f21202c69df478796a5d7dcfe8b07b5"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.405798 18852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9f21202c69df478796a5d7dcfe8b07b5"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.409456 18852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:58:09.411753 18867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.412788 18852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:09.412897 18852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9f21202c69df478796a5d7dcfe8b07b5"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.412982 18852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:09.437187 18852 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:09.437827 18852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:09.437983 18852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.438535 18852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:09.439509 18852 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:09.439559 18852 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.439620 18852 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:09.439646 18852 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.448460 18852 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.131:44389
I20240502 07:58:09.448489 18980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.131:44389 every 8 connection(s)
I20240502 07:58:09.449383 18852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 07:58:09.454087 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 18852
I20240502 07:58:09.454202 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 07:58:09.456467 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.132:0
--local_ip_for_outbound_sockets=127.18.13.132
--webserver_interface=127.18.13.132
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46263
--builtin_ntp_servers=127.18.13.148:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:09.458395 18981 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46263
I20240502 07:58:09.458609 18981 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.459120 18981 heartbeater.cc:511] Master 127.18.13.190:46263 requested a full tablet report, sending...
W20240502 07:58:09.566290 18984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:09.566504 18984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:09.566572 18984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 07:58:09.566736 18529 ts_manager.cc:194] Registered new tserver with Master: 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389)
I20240502 07:58:09.567440 18529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.131:38759
W20240502 07:58:09.569849 18984 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:09.569924 18984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:09.570019 18984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.132
I20240502 07:58:09.573695 18984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.18.13.132
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46263
--never_fsync=true
--heap_profile_path=/tmp/kudu.18984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.132
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:09.574600 18984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:09.575718 18984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:09.584908 18993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.586023 18990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.587103 18991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.588146 18984 server_base.cc:1034] running on GCE node
I20240502 07:58:09.588585 18984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:09.589193 18984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:09.590991 18984 hybrid_clock.cc:648] HybridClock initialized: now 1714636689590971 us; error 30 us; skew 500 ppm
I20240502 07:58:09.593139 18984 webserver.cc:460] Webserver started at http://127.18.13.132:36259/ using document root <none> and password file <none>
I20240502 07:58:09.593710 18984 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.593772 18984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.593950 18984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.595769 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "341d0eb74c3544faaa2633d7aa3f77b3"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.596302 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "341d0eb74c3544faaa2633d7aa3f77b3"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.599599 18984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 07:58:09.601929 18999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.602950 18984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:09.603060 18984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "341d0eb74c3544faaa2633d7aa3f77b3"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.603144 18984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:09.631310 18984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:09.632045 18984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:09.632210 18984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.632752 18984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:09.633625 18984 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:09.633675 18984 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.633736 18984 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:09.633774 18984 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.642772 18984 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.132:35417
I20240502 07:58:09.642804 19112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.132:35417 every 8 connection(s)
I20240502 07:58:09.643697 18984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:58:09.652613 19113 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46263
I20240502 07:58:09.652681 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 18984
I20240502 07:58:09.652776 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 07:58:09.652871 19113 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.653580 19113 heartbeater.cc:511] Master 127.18.13.190:46263 requested a full tablet report, sending...
I20240502 07:58:09.655138 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.133:0
--local_ip_for_outbound_sockets=127.18.13.133
--webserver_interface=127.18.13.133
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46263
--builtin_ntp_servers=127.18.13.148:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:09.767583 18529 ts_manager.cc:194] Registered new tserver with Master: 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417)
W20240502 07:58:09.768047 19117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:09.768316 19117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:09.768385 19117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 07:58:09.768560 18529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.132:56679
W20240502 07:58:09.771821 19117 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:09.771929 19117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:09.772046 19117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.133
I20240502 07:58:09.775698 19117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.18.13.133
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46263
--never_fsync=true
--heap_profile_path=/tmp/kudu.19117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.133
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:09.776657 19117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:09.777792 19117 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:09.787305 19123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.788273 19125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.789147 19122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.790467 19117 server_base.cc:1034] running on GCE node
I20240502 07:58:09.790896 19117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:09.791481 19117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:09.792631 19117 hybrid_clock.cc:648] HybridClock initialized: now 1714636689792605 us; error 48 us; skew 500 ppm
I20240502 07:58:09.794627 19117 webserver.cc:460] Webserver started at http://127.18.13.133:32911/ using document root <none> and password file <none>
I20240502 07:58:09.795213 19117 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.795274 19117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.795454 19117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.797216 19117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d19f943e214647a3850f51a324017662"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.797698 19117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d19f943e214647a3850f51a324017662"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.801357 19117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240502 07:58:09.803602 19131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.804670 19117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 07:58:09.804781 19117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d19f943e214647a3850f51a324017662"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.804867 19117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:09.824818 19117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:09.825433 19117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:09.825598 19117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.826097 19117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:09.827582 19117 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:09.827636 19117 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:09.827701 19117 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:09.827728 19117 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.836336 19117 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.133:40945
I20240502 07:58:09.836382 19244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.133:40945 every 8 connection(s)
I20240502 07:58:09.837334 19117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:58:09.841547 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 19117
I20240502 07:58:09.841653 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 07:58:09.843945 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.134:0
--local_ip_for_outbound_sockets=127.18.13.134
--webserver_interface=127.18.13.134
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46263
--builtin_ntp_servers=127.18.13.148:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:09.846714 19245 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46263
I20240502 07:58:09.846971 19245 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.847659 19245 heartbeater.cc:511] Master 127.18.13.190:46263 requested a full tablet report, sending...
W20240502 07:58:09.967528 19248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:09.967840 19248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:09.967975 19248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 07:58:09.972043 18529 ts_manager.cc:194] Registered new tserver with Master: d19f943e214647a3850f51a324017662 (127.18.13.133:40945)
W20240502 07:58:09.972291 19248 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:09.972378 19248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:09.972527 19248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.134
I20240502 07:58:09.972872 18529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.133:34669
I20240502 07:58:09.976317 19248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.18.13.134
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46263
--never_fsync=true
--heap_profile_path=/tmp/kudu.19248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.134
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:09.977245 19248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:09.978354 19248 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:09.987709 19255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.988685 19254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.989995 19257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.991245 19248 server_base.cc:1034] running on GCE node
I20240502 07:58:09.991822 19248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:09.992537 19248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:09.993726 19248 hybrid_clock.cc:648] HybridClock initialized: now 1714636689993701 us; error 40 us; skew 500 ppm
I20240502 07:58:09.995983 19248 webserver.cc:460] Webserver started at http://127.18.13.134:41981/ using document root <none> and password file <none>
I20240502 07:58:09.996659 19248 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.996747 19248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.996950 19248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.998804 19248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "345ab146afdf4d39a1d8583881c06743"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:09.999428 19248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "345ab146afdf4d39a1d8583881c06743"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:10.003420 19248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240502 07:58:10.005947 19263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:10.007166 19248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:10.007278 19248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "345ab146afdf4d39a1d8583881c06743"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-f46z"
I20240502 07:58:10.007372 19248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:10.022843 19248 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:10.023460 19248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:10.023614 19248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:10.024188 19248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:10.025056 19248 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:10.025115 19248 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:10.025173 19248 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:10.025198 19248 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:10.034464 19248 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.134:44647
I20240502 07:58:10.034493 19376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.134:44647 every 8 connection(s)
I20240502 07:58:10.035558 19248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 07:58:10.040786 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 19248
I20240502 07:58:10.040889 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 07:58:10.044780 19377 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46263
I20240502 07:58:10.045001 19377 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:10.045519 19377 heartbeater.cc:511] Master 127.18.13.190:46263 requested a full tablet report, sending...
I20240502 07:58:10.107139 18717 heartbeater.cc:503] Master 127.18.13.190:46263 was elected leader, sending a full tablet report...
I20240502 07:58:10.179718 18529 ts_manager.cc:194] Registered new tserver with Master: 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647)
I20240502 07:58:10.180444 18529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.134:48353
I20240502 07:58:10.188589 18486 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 07:58:10.201365 18486 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 07:58:10.309626 18849 heartbeater.cc:503] Master 127.18.13.190:46263 was elected leader, sending a full tablet report...
I20240502 07:58:10.569658 18981 heartbeater.cc:503] Master 127.18.13.190:46263 was elected leader, sending a full tablet report...
I20240502 07:58:10.770906 19113 heartbeater.cc:503] Master 127.18.13.190:46263 was elected leader, sending a full tablet report...
I20240502 07:58:10.975188 19245 heartbeater.cc:503] Master 127.18.13.190:46263 was elected leader, sending a full tablet report...
I20240502 07:58:11.182912 19377 heartbeater.cc:503] Master 127.18.13.190:46263 was elected leader, sending a full tablet report...
I20240502 07:58:13.954027 18529 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45150:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:13.974866 18783 tablet_service.cc:1467] Processing CreateTablet for tablet 9656cbeef84441f3ba7be5301d1ba339 (DEFAULT_TABLE table=rebalance_test_table_0 [id=76a14fc077a0476ea2cbd4e943ecd17f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:13.974932 19047 tablet_service.cc:1467] Processing CreateTablet for tablet 9656cbeef84441f3ba7be5301d1ba339 (DEFAULT_TABLE table=rebalance_test_table_0 [id=76a14fc077a0476ea2cbd4e943ecd17f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:13.975916 18783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9656cbeef84441f3ba7be5301d1ba339. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:13.976044 19047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9656cbeef84441f3ba7be5301d1ba339. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:13.978267 18650 tablet_service.cc:1467] Processing CreateTablet for tablet fe229d37d9f44877a5b2720f3a7ef6cd (DEFAULT_TABLE table=rebalance_test_table_0 [id=76a14fc077a0476ea2cbd4e943ecd17f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:13.979092 18650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe229d37d9f44877a5b2720f3a7ef6cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:13.980122 19179 tablet_service.cc:1467] Processing CreateTablet for tablet 9656cbeef84441f3ba7be5301d1ba339 (DEFAULT_TABLE table=rebalance_test_table_0 [id=76a14fc077a0476ea2cbd4e943ecd17f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:13.980285 19310 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1cbd7ae66a4d25a82a5020b1d70fb6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=76a14fc077a0476ea2cbd4e943ecd17f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:13.980856 19179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9656cbeef84441f3ba7be5301d1ba339. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:13.981040 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1cbd7ae66a4d25a82a5020b1d70fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:13.978267 18651 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1cbd7ae66a4d25a82a5020b1d70fb6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=76a14fc077a0476ea2cbd4e943ecd17f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:13.984210 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1cbd7ae66a4d25a82a5020b1d70fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:13.984175 18915 tablet_service.cc:1467] Processing CreateTablet for tablet 6f1cbd7ae66a4d25a82a5020b1d70fb6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=76a14fc077a0476ea2cbd4e943ecd17f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:13.984910 18915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f1cbd7ae66a4d25a82a5020b1d70fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:13.987504 19178 tablet_service.cc:1467] Processing CreateTablet for tablet fe229d37d9f44877a5b2720f3a7ef6cd (DEFAULT_TABLE table=rebalance_test_table_0 [id=76a14fc077a0476ea2cbd4e943ecd17f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:13.987576 19403 tablet_bootstrap.cc:492] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:13.987756 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe229d37d9f44877a5b2720f3a7ef6cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:13.989192 19404 tablet_bootstrap.cc:492] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:13.989734 19403 tablet_bootstrap.cc:654] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:13.990564 19403 log.cc:824] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 2f6bc76c600c413d9f72526563d3234a: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:13.991076 19404 tablet_bootstrap.cc:654] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:13.991387 18914 tablet_service.cc:1467] Processing CreateTablet for tablet fe229d37d9f44877a5b2720f3a7ef6cd (DEFAULT_TABLE table=rebalance_test_table_0 [id=76a14fc077a0476ea2cbd4e943ecd17f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:13.991643 18914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe229d37d9f44877a5b2720f3a7ef6cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:13.991948 19404 log.cc:824] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:13.992298 19403 tablet_bootstrap.cc:492] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:13.992466 19403 ts_tablet_manager.cc:1367] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 07:58:13.994484 19402 tablet_bootstrap.cc:492] T 9656cbeef84441f3ba7be5301d1ba339 P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:13.995846 19405 tablet_bootstrap.cc:492] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:13.996470 19402 tablet_bootstrap.cc:654] T 9656cbeef84441f3ba7be5301d1ba339 P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:13.996989 19403 raft_consensus.cc:348] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:13.997313 19403 raft_consensus.cc:374] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:13.997321 19402 log.cc:824] T 9656cbeef84441f3ba7be5301d1ba339 P 341d0eb74c3544faaa2633d7aa3f77b3: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:13.997457 19403 raft_consensus.cc:729] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:13.997656 19404 tablet_bootstrap.cc:492] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:13.997855 19404 ts_tablet_manager.cc:1367] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 07:58:13.997997 19405 tablet_bootstrap.cc:654] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:13.998004 19403 consensus_queue.cc:260] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:13.998750 19405 log.cc:824] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:13.998982 19403 ts_tablet_manager.cc:1398] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 07:58:13.998996 19402 tablet_bootstrap.cc:492] T 9656cbeef84441f3ba7be5301d1ba339 P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:13.999155 19402 ts_tablet_manager.cc:1367] T 9656cbeef84441f3ba7be5301d1ba339 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 07:58:13.999392 19403 tablet_bootstrap.cc:492] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:14.000438 19403 tablet_bootstrap.cc:654] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.001529 19408 tablet_bootstrap.cc:492] T fe229d37d9f44877a5b2720f3a7ef6cd P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:14.001672 19403 tablet_bootstrap.cc:492] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:14.001825 19403 ts_tablet_manager.cc:1367] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:14.002699 19402 raft_consensus.cc:348] T 9656cbeef84441f3ba7be5301d1ba339 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.002935 19402 raft_consensus.cc:374] T 9656cbeef84441f3ba7be5301d1ba339 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.003026 19402 raft_consensus.cc:729] T 9656cbeef84441f3ba7be5301d1ba339 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.002406 19403 raft_consensus.cc:348] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.003219 19403 raft_consensus.cc:374] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.003283 19403 raft_consensus.cc:729] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.003377 19409 tablet_bootstrap.cc:492] T fe229d37d9f44877a5b2720f3a7ef6cd P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:14.003440 19403 consensus_queue.cc:260] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.003505 19402 consensus_queue.cc:260] T 9656cbeef84441f3ba7be5301d1ba339 P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.003742 19403 ts_tablet_manager.cc:1398] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:14.004402 19402 ts_tablet_manager.cc:1398] T 9656cbeef84441f3ba7be5301d1ba339 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 07:58:14.004869 19408 tablet_bootstrap.cc:654] T fe229d37d9f44877a5b2720f3a7ef6cd P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.005903 19404 raft_consensus.cc:348] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.006067 19404 raft_consensus.cc:374] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.006114 19404 raft_consensus.cc:729] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.006528 19404 consensus_queue.cc:260] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.006994 19409 tablet_bootstrap.cc:654] T fe229d37d9f44877a5b2720f3a7ef6cd P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.007298 19404 ts_tablet_manager.cc:1398] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240502 07:58:14.008285 19408 log.cc:824] T fe229d37d9f44877a5b2720f3a7ef6cd P d19f943e214647a3850f51a324017662: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:14.008656 19409 log.cc:824] T fe229d37d9f44877a5b2720f3a7ef6cd P 9f21202c69df478796a5d7dcfe8b07b5: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:14.010403 19405 tablet_bootstrap.cc:492] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:14.010637 19405 ts_tablet_manager.cc:1367] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 07:58:14.010749 19408 tablet_bootstrap.cc:492] T fe229d37d9f44877a5b2720f3a7ef6cd P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:14.010912 19408 ts_tablet_manager.cc:1367] T fe229d37d9f44877a5b2720f3a7ef6cd P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 07:58:14.011508 19409 tablet_bootstrap.cc:492] T fe229d37d9f44877a5b2720f3a7ef6cd P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:14.011664 19409 ts_tablet_manager.cc:1367] T fe229d37d9f44877a5b2720f3a7ef6cd P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 07:58:14.013666 19408 raft_consensus.cc:348] T fe229d37d9f44877a5b2720f3a7ef6cd P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.013832 19408 raft_consensus.cc:374] T fe229d37d9f44877a5b2720f3a7ef6cd P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.013906 19408 raft_consensus.cc:729] T fe229d37d9f44877a5b2720f3a7ef6cd P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.014281 19408 consensus_queue.cc:260] T fe229d37d9f44877a5b2720f3a7ef6cd P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.014334 19405 raft_consensus.cc:348] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.014537 19405 raft_consensus.cc:374] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.014605 19405 raft_consensus.cc:729] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.014999 19405 consensus_queue.cc:260] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.015362 19409 raft_consensus.cc:348] T fe229d37d9f44877a5b2720f3a7ef6cd P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.015568 19408 ts_tablet_manager.cc:1398] T fe229d37d9f44877a5b2720f3a7ef6cd P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240502 07:58:14.015573 19409 raft_consensus.cc:374] T fe229d37d9f44877a5b2720f3a7ef6cd P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.015633 19409 raft_consensus.cc:729] T fe229d37d9f44877a5b2720f3a7ef6cd P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.015839 19405 ts_tablet_manager.cc:1398] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20240502 07:58:14.016044 19409 consensus_queue.cc:260] T fe229d37d9f44877a5b2720f3a7ef6cd P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.016131 19408 tablet_bootstrap.cc:492] T 9656cbeef84441f3ba7be5301d1ba339 P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:14.016904 19409 ts_tablet_manager.cc:1398] T fe229d37d9f44877a5b2720f3a7ef6cd P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 07:58:14.017242 19409 tablet_bootstrap.cc:492] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:14.018234 19409 tablet_bootstrap.cc:654] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.018721 19408 tablet_bootstrap.cc:654] T 9656cbeef84441f3ba7be5301d1ba339 P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.019537 19409 tablet_bootstrap.cc:492] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:14.019630 19409 ts_tablet_manager.cc:1367] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:14.020109 19408 tablet_bootstrap.cc:492] T 9656cbeef84441f3ba7be5301d1ba339 P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:14.020188 19408 ts_tablet_manager.cc:1367] T 9656cbeef84441f3ba7be5301d1ba339 P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:14.020184 19409 raft_consensus.cc:348] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.020293 19409 raft_consensus.cc:374] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.020330 19409 raft_consensus.cc:729] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.020449 19409 consensus_queue.cc:260] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.020681 19409 ts_tablet_manager.cc:1398] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:14.020678 19408 raft_consensus.cc:348] T 9656cbeef84441f3ba7be5301d1ba339 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.020817 19408 raft_consensus.cc:374] T 9656cbeef84441f3ba7be5301d1ba339 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.020865 19408 raft_consensus.cc:729] T 9656cbeef84441f3ba7be5301d1ba339 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.020995 19408 consensus_queue.cc:260] T 9656cbeef84441f3ba7be5301d1ba339 P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.021243 19408 ts_tablet_manager.cc:1398] T 9656cbeef84441f3ba7be5301d1ba339 P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240502 07:58:14.039417 19378 tablet.cc:2369] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 07:58:14.056838 18718 tablet.cc:2369] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 2f6bc76c600c413d9f72526563d3234a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:14.059054 19411 raft_consensus.cc:482] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:14.059211 19411 raft_consensus.cc:504] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.060338 19411 leader_election.cc:288] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389), d19f943e214647a3850f51a324017662 (127.18.13.133:40945)
I20240502 07:58:14.064227 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe229d37d9f44877a5b2720f3a7ef6cd" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662" is_pre_election: true
I20240502 07:58:14.064359 18935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe229d37d9f44877a5b2720f3a7ef6cd" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5" is_pre_election: true
I20240502 07:58:14.064502 19199 raft_consensus.cc:2454] T fe229d37d9f44877a5b2720f3a7ef6cd P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 0.
I20240502 07:58:14.064641 18935 raft_consensus.cc:2454] T fe229d37d9f44877a5b2720f3a7ef6cd P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 0.
I20240502 07:58:14.064926 18604 leader_election.cc:302] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:14.065125 19411 raft_consensus.cc:2781] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:14.065193 19411 raft_consensus.cc:482] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:14.065239 19411 raft_consensus.cc:3037] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.066305 19411 raft_consensus.cc:504] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.066582 19411 leader_election.cc:288] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 election: Requested vote from peers 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389), d19f943e214647a3850f51a324017662 (127.18.13.133:40945)
I20240502 07:58:14.066874 18935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe229d37d9f44877a5b2720f3a7ef6cd" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5"
I20240502 07:58:14.066887 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe229d37d9f44877a5b2720f3a7ef6cd" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662"
I20240502 07:58:14.066978 18935 raft_consensus.cc:3037] T fe229d37d9f44877a5b2720f3a7ef6cd P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.067001 19199 raft_consensus.cc:3037] T fe229d37d9f44877a5b2720f3a7ef6cd P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.067827 19199 raft_consensus.cc:2454] T fe229d37d9f44877a5b2720f3a7ef6cd P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 1.
I20240502 07:58:14.067843 18935 raft_consensus.cc:2454] T fe229d37d9f44877a5b2720f3a7ef6cd P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 1.
I20240502 07:58:14.068064 18604 leader_election.cc:302] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:14.068276 19411 raft_consensus.cc:2781] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:14.068480 19411 raft_consensus.cc:686] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [term 1 LEADER]: Becoming Leader. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Running, Role: LEADER
I20240502 07:58:14.068769 19411 consensus_queue.cc:237] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.071535 18529 catalog_manager.cc:5484] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f6bc76c600c413d9f72526563d3234a (127.18.13.129). New cstate: current_term: 1 leader_uuid: "2f6bc76c600c413d9f72526563d3234a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:14.091255 19246 tablet.cc:2369] T 9656cbeef84441f3ba7be5301d1ba339 P d19f943e214647a3850f51a324017662: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 07:58:14.147678 19114 tablet.cc:2369] T 9656cbeef84441f3ba7be5301d1ba339 P 341d0eb74c3544faaa2633d7aa3f77b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 07:58:14.203680 18982 tablet.cc:2369] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 9f21202c69df478796a5d7dcfe8b07b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:14.236358 19415 raft_consensus.cc:482] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:14.236575 19415 raft_consensus.cc:504] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.237720 19415 leader_election.cc:288] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523), 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389)
I20240502 07:58:14.242779 18935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1cbd7ae66a4d25a82a5020b1d70fb6" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5" is_pre_election: true
I20240502 07:58:14.242935 18935 raft_consensus.cc:2454] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 0.
I20240502 07:58:14.242920 18671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1cbd7ae66a4d25a82a5020b1d70fb6" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a" is_pre_election: true
I20240502 07:58:14.243155 18671 raft_consensus.cc:2454] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 0.
I20240502 07:58:14.243290 19265 leader_election.cc:302] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345ab146afdf4d39a1d8583881c06743, 9f21202c69df478796a5d7dcfe8b07b5; no voters:
I20240502 07:58:14.243542 19415 raft_consensus.cc:2781] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:14.243605 19415 raft_consensus.cc:482] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:14.243633 19415 raft_consensus.cc:3037] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.244495 19415 raft_consensus.cc:504] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.244788 19415 leader_election.cc:288] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 election: Requested vote from peers 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523), 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389)
I20240502 07:58:14.245131 18671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1cbd7ae66a4d25a82a5020b1d70fb6" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a"
I20240502 07:58:14.245173 18935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f1cbd7ae66a4d25a82a5020b1d70fb6" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5"
I20240502 07:58:14.245236 18671 raft_consensus.cc:3037] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.245301 18935 raft_consensus.cc:3037] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.246045 18671 raft_consensus.cc:2454] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 1.
I20240502 07:58:14.246104 18935 raft_consensus.cc:2454] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 1.
W20240502 07:58:14.246265 18850 tablet.cc:2369] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:14.246418 19266 leader_election.cc:302] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, 345ab146afdf4d39a1d8583881c06743; no voters:
I20240502 07:58:14.246606 19415 raft_consensus.cc:2781] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:14.246819 19415 raft_consensus.cc:686] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [term 1 LEADER]: Becoming Leader. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Running, Role: LEADER
I20240502 07:58:14.247107 19415 consensus_queue.cc:237] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.250238 18529 catalog_manager.cc:5484] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 reported cstate change: term changed from 0 to 1, leader changed from <none> to 345ab146afdf4d39a1d8583881c06743 (127.18.13.134). New cstate: current_term: 1 leader_uuid: "345ab146afdf4d39a1d8583881c06743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:14.255600 19419 raft_consensus.cc:482] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:14.255775 19419 raft_consensus.cc:504] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.256780 19419 leader_election.cc:288] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417), d19f943e214647a3850f51a324017662 (127.18.13.133:40945)
I20240502 07:58:14.261358 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9656cbeef84441f3ba7be5301d1ba339" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662" is_pre_election: true
I20240502 07:58:14.261529 19199 raft_consensus.cc:2454] T 9656cbeef84441f3ba7be5301d1ba339 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 0.
I20240502 07:58:14.261515 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9656cbeef84441f3ba7be5301d1ba339" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" is_pre_election: true
I20240502 07:58:14.261772 19067 raft_consensus.cc:2454] T 9656cbeef84441f3ba7be5301d1ba339 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 0.
I20240502 07:58:14.261888 18736 leader_election.cc:302] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4135e6d90aec4830b08bcb7e8dc50260, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:14.262157 19419 raft_consensus.cc:2781] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:14.262233 19419 raft_consensus.cc:482] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:14.262279 19419 raft_consensus.cc:3037] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.263103 19419 raft_consensus.cc:504] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.263417 19419 leader_election.cc:288] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 election: Requested vote from peers 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417), d19f943e214647a3850f51a324017662 (127.18.13.133:40945)
I20240502 07:58:14.263705 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9656cbeef84441f3ba7be5301d1ba339" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3"
I20240502 07:58:14.263746 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9656cbeef84441f3ba7be5301d1ba339" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662"
I20240502 07:58:14.263825 19067 raft_consensus.cc:3037] T 9656cbeef84441f3ba7be5301d1ba339 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.263887 19199 raft_consensus.cc:3037] T 9656cbeef84441f3ba7be5301d1ba339 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.264644 19199 raft_consensus.cc:2454] T 9656cbeef84441f3ba7be5301d1ba339 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 1.
I20240502 07:58:14.264801 19067 raft_consensus.cc:2454] T 9656cbeef84441f3ba7be5301d1ba339 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 1.
I20240502 07:58:14.264911 18736 leader_election.cc:302] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4135e6d90aec4830b08bcb7e8dc50260, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:14.265080 19419 raft_consensus.cc:2781] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:14.265336 19419 raft_consensus.cc:686] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 LEADER]: Becoming Leader. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Running, Role: LEADER
I20240502 07:58:14.265657 19419 consensus_queue.cc:237] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:14.268829 18529 catalog_manager.cc:5484] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "4135e6d90aec4830b08bcb7e8dc50260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:14.391567 19435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:14.391834 19435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:14.394877 19435 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 07:58:14.513542 19199 raft_consensus.cc:1261] T fe229d37d9f44877a5b2720f3a7ef6cd P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Refusing update from remote peer 2f6bc76c600c413d9f72526563d3234a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:14.513598 18935 raft_consensus.cc:1261] T fe229d37d9f44877a5b2720f3a7ef6cd P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Refusing update from remote peer 2f6bc76c600c413d9f72526563d3234a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:14.514216 19411 consensus_queue.cc:1035] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:14.514376 19425 consensus_queue.cc:1035] T fe229d37d9f44877a5b2720f3a7ef6cd P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:14.516355 18934 raft_consensus.cc:1261] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Refusing update from remote peer 345ab146afdf4d39a1d8583881c06743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:14.516875 18671 raft_consensus.cc:1261] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Refusing update from remote peer 345ab146afdf4d39a1d8583881c06743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:14.517354 19429 consensus_queue.cc:1035] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:14.517758 19429 consensus_queue.cc:1035] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:14.518540 19067 raft_consensus.cc:1261] T 9656cbeef84441f3ba7be5301d1ba339 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Refusing update from remote peer 4135e6d90aec4830b08bcb7e8dc50260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:14.519198 19419 consensus_queue.cc:1035] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:14.523958 19198 raft_consensus.cc:1261] T 9656cbeef84441f3ba7be5301d1ba339 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Refusing update from remote peer 4135e6d90aec4830b08bcb7e8dc50260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:14.524479 19419 consensus_queue.cc:1035] T 9656cbeef84441f3ba7be5301d1ba339 P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:14.526280 19471 mvcc.cc:205] Tried to move back new op lower bound from 7023151900732223488 to 7023151899638513664. Current Snapshot: MvccSnapshot[applied={T|T < 7023151900732223488}]
I20240502 07:58:14.531540 19470 mvcc.cc:205] Tried to move back new op lower bound from 7023151900732223488 to 7023151899638513664. Current Snapshot: MvccSnapshot[applied={T|T < 7023151900732223488}]
I20240502 07:58:14.535105 19469 mvcc.cc:205] Tried to move back new op lower bound from 7023151900738416640 to 7023151899714613248. Current Snapshot: MvccSnapshot[applied={T|T < 7023151900738416640}]
I20240502 07:58:14.550885 19482 mvcc.cc:205] Tried to move back new op lower bound from 7023151900720259072 to 7023151898907832320. Current Snapshot: MvccSnapshot[applied={T|T < 7023151900720259072}]
INSERT report
rows total: 2000
time total: 131.087 ms
time per row: 0.0655434 ms
I20240502 07:58:14.578621 19474 mvcc.cc:205] Tried to move back new op lower bound from 7023151900720259072 to 7023151898907832320. Current Snapshot: MvccSnapshot[applied={T|T < 7023151900720259072}]
I20240502 07:58:14.581808 18526 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45150:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:14.592970 18783 tablet_service.cc:1467] Processing CreateTablet for tablet f0b7aba425554d5f98024438383bbf37 (DEFAULT_TABLE table=rebalance_test_table_1 [id=630d237543764b7594ce0368fcb4b77a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.593039 19311 tablet_service.cc:1467] Processing CreateTablet for tablet fce88a610b8747d090d7fdda3d782abb (DEFAULT_TABLE table=rebalance_test_table_1 [id=630d237543764b7594ce0368fcb4b77a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.593281 18783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b7aba425554d5f98024438383bbf37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.593384 19311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce88a610b8747d090d7fdda3d782abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.595589 18782 tablet_service.cc:1467] Processing CreateTablet for tablet fce88a610b8747d090d7fdda3d782abb (DEFAULT_TABLE table=rebalance_test_table_1 [id=630d237543764b7594ce0368fcb4b77a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.595809 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce88a610b8747d090d7fdda3d782abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.597877 19310 tablet_service.cc:1467] Processing CreateTablet for tablet f0b7aba425554d5f98024438383bbf37 (DEFAULT_TABLE table=rebalance_test_table_1 [id=630d237543764b7594ce0368fcb4b77a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.598120 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b7aba425554d5f98024438383bbf37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.599463 19503 tablet_bootstrap.cc:492] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:14.600155 19179 tablet_service.cc:1467] Processing CreateTablet for tablet 112385d70673444fb1c253255d47d8f8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=630d237543764b7594ce0368fcb4b77a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.600400 19179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 112385d70673444fb1c253255d47d8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.601917 19503 tablet_bootstrap.cc:654] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.602600 19505 tablet_bootstrap.cc:492] T 112385d70673444fb1c253255d47d8f8 P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:14.603271 19047 tablet_service.cc:1467] Processing CreateTablet for tablet f0b7aba425554d5f98024438383bbf37 (DEFAULT_TABLE table=rebalance_test_table_1 [id=630d237543764b7594ce0368fcb4b77a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.603363 19046 tablet_service.cc:1467] Processing CreateTablet for tablet fce88a610b8747d090d7fdda3d782abb (DEFAULT_TABLE table=rebalance_test_table_1 [id=630d237543764b7594ce0368fcb4b77a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.603535 19047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b7aba425554d5f98024438383bbf37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.603678 19046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce88a610b8747d090d7fdda3d782abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.604686 19505 tablet_bootstrap.cc:654] T 112385d70673444fb1c253255d47d8f8 P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.606467 18914 tablet_service.cc:1467] Processing CreateTablet for tablet 112385d70673444fb1c253255d47d8f8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=630d237543764b7594ce0368fcb4b77a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.606756 18914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 112385d70673444fb1c253255d47d8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.606940 18650 tablet_service.cc:1467] Processing CreateTablet for tablet 112385d70673444fb1c253255d47d8f8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=630d237543764b7594ce0368fcb4b77a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.607273 18650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 112385d70673444fb1c253255d47d8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.610437 19509 tablet_bootstrap.cc:492] T 112385d70673444fb1c253255d47d8f8 P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:14.611557 19509 tablet_bootstrap.cc:654] T 112385d70673444fb1c253255d47d8f8 P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.612601 19504 tablet_bootstrap.cc:492] T fce88a610b8747d090d7fdda3d782abb P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:14.612860 19509 tablet_bootstrap.cc:492] T 112385d70673444fb1c253255d47d8f8 P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:14.612970 19509 ts_tablet_manager.cc:1367] T 112385d70673444fb1c253255d47d8f8 P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:14.613407 19504 tablet_bootstrap.cc:654] T fce88a610b8747d090d7fdda3d782abb P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.613706 19509 raft_consensus.cc:348] T 112385d70673444fb1c253255d47d8f8 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.613842 19509 raft_consensus.cc:374] T 112385d70673444fb1c253255d47d8f8 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.613871 19509 raft_consensus.cc:729] T 112385d70673444fb1c253255d47d8f8 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.614014 19508 tablet_bootstrap.cc:492] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:14.614012 19509 consensus_queue.cc:260] T 112385d70673444fb1c253255d47d8f8 P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.614262 19509 ts_tablet_manager.cc:1398] T 112385d70673444fb1c253255d47d8f8 P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:14.614744 19508 tablet_bootstrap.cc:654] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.615286 19504 tablet_bootstrap.cc:492] T fce88a610b8747d090d7fdda3d782abb P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:14.615370 19504 ts_tablet_manager.cc:1367] T fce88a610b8747d090d7fdda3d782abb P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:58:14.615542 19507 tablet_bootstrap.cc:492] T f0b7aba425554d5f98024438383bbf37 P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:14.615859 19503 tablet_bootstrap.cc:492] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:14.616000 19503 ts_tablet_manager.cc:1367] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 07:58:14.615979 19504 raft_consensus.cc:348] T fce88a610b8747d090d7fdda3d782abb P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.616117 19504 raft_consensus.cc:374] T fce88a610b8747d090d7fdda3d782abb P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.616165 19504 raft_consensus.cc:729] T fce88a610b8747d090d7fdda3d782abb P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.616312 19504 consensus_queue.cc:260] T fce88a610b8747d090d7fdda3d782abb P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.616674 19504 ts_tablet_manager.cc:1398] T fce88a610b8747d090d7fdda3d782abb P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:14.616698 19503 raft_consensus.cc:348] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.616848 19503 raft_consensus.cc:374] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.616873 19504 tablet_bootstrap.cc:492] T f0b7aba425554d5f98024438383bbf37 P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:14.616900 19503 raft_consensus.cc:729] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.617069 19503 consensus_queue.cc:260] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.617164 19505 tablet_bootstrap.cc:492] T 112385d70673444fb1c253255d47d8f8 P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:14.617269 19505 ts_tablet_manager.cc:1367] T 112385d70673444fb1c253255d47d8f8 P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240502 07:58:14.617430 19503 ts_tablet_manager.cc:1398] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:14.617589 19503 tablet_bootstrap.cc:492] T fce88a610b8747d090d7fdda3d782abb P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:14.617769 19505 raft_consensus.cc:348] T 112385d70673444fb1c253255d47d8f8 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.617882 19505 raft_consensus.cc:374] T 112385d70673444fb1c253255d47d8f8 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.617939 19505 raft_consensus.cc:729] T 112385d70673444fb1c253255d47d8f8 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.618026 19504 tablet_bootstrap.cc:654] T f0b7aba425554d5f98024438383bbf37 P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.618088 19505 consensus_queue.cc:260] T 112385d70673444fb1c253255d47d8f8 P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.618479 19505 ts_tablet_manager.cc:1398] T 112385d70673444fb1c253255d47d8f8 P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:14.618638 19503 tablet_bootstrap.cc:654] T fce88a610b8747d090d7fdda3d782abb P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.618916 19507 tablet_bootstrap.cc:654] T f0b7aba425554d5f98024438383bbf37 P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.619346 19508 tablet_bootstrap.cc:492] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:14.619447 19508 ts_tablet_manager.cc:1367] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 07:58:14.620182 19507 tablet_bootstrap.cc:492] T f0b7aba425554d5f98024438383bbf37 P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:14.620272 19507 ts_tablet_manager.cc:1367] T f0b7aba425554d5f98024438383bbf37 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:58:14.620584 19504 tablet_bootstrap.cc:492] T f0b7aba425554d5f98024438383bbf37 P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:14.620652 19504 ts_tablet_manager.cc:1367] T f0b7aba425554d5f98024438383bbf37 P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:14.620800 19507 raft_consensus.cc:348] T f0b7aba425554d5f98024438383bbf37 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.620889 19507 raft_consensus.cc:374] T f0b7aba425554d5f98024438383bbf37 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.620931 19507 raft_consensus.cc:729] T f0b7aba425554d5f98024438383bbf37 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.620966 19503 tablet_bootstrap.cc:492] T fce88a610b8747d090d7fdda3d782abb P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:14.621034 19503 ts_tablet_manager.cc:1367] T fce88a610b8747d090d7fdda3d782abb P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:58:14.621058 19504 raft_consensus.cc:348] T f0b7aba425554d5f98024438383bbf37 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.621049 19507 consensus_queue.cc:260] T f0b7aba425554d5f98024438383bbf37 P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.621133 19504 raft_consensus.cc:374] T f0b7aba425554d5f98024438383bbf37 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.621165 19504 raft_consensus.cc:729] T f0b7aba425554d5f98024438383bbf37 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.621271 19504 consensus_queue.cc:260] T f0b7aba425554d5f98024438383bbf37 P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.621318 19507 ts_tablet_manager.cc:1398] T f0b7aba425554d5f98024438383bbf37 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:14.621469 19507 tablet_bootstrap.cc:492] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:14.621433 19503 raft_consensus.cc:348] T fce88a610b8747d090d7fdda3d782abb P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.621528 19504 ts_tablet_manager.cc:1398] T f0b7aba425554d5f98024438383bbf37 P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:14.621541 19503 raft_consensus.cc:374] T fce88a610b8747d090d7fdda3d782abb P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.621572 19503 raft_consensus.cc:729] T fce88a610b8747d090d7fdda3d782abb P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.621680 19503 consensus_queue.cc:260] T fce88a610b8747d090d7fdda3d782abb P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.621933 19503 ts_tablet_manager.cc:1398] T fce88a610b8747d090d7fdda3d782abb P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:14.621891 19508 raft_consensus.cc:348] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.622000 19508 raft_consensus.cc:374] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.622030 19508 raft_consensus.cc:729] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.622144 19508 consensus_queue.cc:260] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.622347 19507 tablet_bootstrap.cc:654] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.622514 19508 ts_tablet_manager.cc:1398] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:58:14.623374 19507 tablet_bootstrap.cc:492] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:14.623461 19507 ts_tablet_manager.cc:1367] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:14.623910 19507 raft_consensus.cc:348] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.624033 19507 raft_consensus.cc:374] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.624064 19507 raft_consensus.cc:729] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.624189 19507 consensus_queue.cc:260] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.624459 19507 ts_tablet_manager.cc:1398] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:14.647547 19419 raft_consensus.cc:482] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:14.647678 19419 raft_consensus.cc:504] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.648191 19419 leader_election.cc:288] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647), 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417)
I20240502 07:58:14.648731 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b7aba425554d5f98024438383bbf37" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" is_pre_election: true
I20240502 07:58:14.648854 19067 raft_consensus.cc:2454] T f0b7aba425554d5f98024438383bbf37 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 0.
I20240502 07:58:14.649148 18738 leader_election.cc:302] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:14.649343 19419 raft_consensus.cc:2781] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:14.649398 19419 raft_consensus.cc:482] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:14.649431 19419 raft_consensus.cc:3037] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.650419 19419 raft_consensus.cc:504] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.650741 19419 leader_election.cc:288] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 election: Requested vote from peers 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647), 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417)
I20240502 07:58:14.651142 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b7aba425554d5f98024438383bbf37" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3"
I20240502 07:58:14.651242 19067 raft_consensus.cc:3037] T f0b7aba425554d5f98024438383bbf37 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.652089 19067 raft_consensus.cc:2454] T f0b7aba425554d5f98024438383bbf37 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 1.
I20240502 07:58:14.652359 18738 leader_election.cc:302] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:14.652495 19419 raft_consensus.cc:2781] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:14.652550 19419 raft_consensus.cc:686] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 LEADER]: Becoming Leader. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Running, Role: LEADER
I20240502 07:58:14.652664 19419 consensus_queue.cc:237] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.652755 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b7aba425554d5f98024438383bbf37" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743" is_pre_election: true
I20240502 07:58:14.653026 19331 raft_consensus.cc:2454] T f0b7aba425554d5f98024438383bbf37 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 0.
I20240502 07:58:14.652758 19330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b7aba425554d5f98024438383bbf37" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743"
I20240502 07:58:14.653342 19330 raft_consensus.cc:3037] T f0b7aba425554d5f98024438383bbf37 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.654444 19330 raft_consensus.cc:2454] T f0b7aba425554d5f98024438383bbf37 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 1.
I20240502 07:58:14.654554 18526 catalog_manager.cc:5484] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "4135e6d90aec4830b08bcb7e8dc50260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:14.744112 19489 raft_consensus.cc:482] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:14.744259 19489 raft_consensus.cc:504] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.745364 19489 leader_election.cc:288] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999), 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647)
I20240502 07:58:14.749477 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce88a610b8747d090d7fdda3d782abb" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260" is_pre_election: true
I20240502 07:58:14.749706 18803 raft_consensus.cc:2454] T fce88a610b8747d090d7fdda3d782abb P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 0.
I20240502 07:58:14.749840 19330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce88a610b8747d090d7fdda3d782abb" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743" is_pre_election: true
I20240502 07:58:14.750051 19330 raft_consensus.cc:2454] T fce88a610b8747d090d7fdda3d782abb P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 0.
I20240502 07:58:14.750123 19000 leader_election.cc:302] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:14.750372 19489 raft_consensus.cc:2781] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:14.750437 19489 raft_consensus.cc:482] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:14.750496 19489 raft_consensus.cc:3037] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.751353 19489 raft_consensus.cc:504] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.751680 19489 leader_election.cc:288] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 election: Requested vote from peers 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999), 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647)
I20240502 07:58:14.752002 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce88a610b8747d090d7fdda3d782abb" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260"
I20240502 07:58:14.752069 19330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce88a610b8747d090d7fdda3d782abb" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743"
I20240502 07:58:14.752139 18803 raft_consensus.cc:3037] T fce88a610b8747d090d7fdda3d782abb P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.752151 19330 raft_consensus.cc:3037] T fce88a610b8747d090d7fdda3d782abb P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.752837 19330 raft_consensus.cc:2454] T fce88a610b8747d090d7fdda3d782abb P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 1.
I20240502 07:58:14.752960 18803 raft_consensus.cc:2454] T fce88a610b8747d090d7fdda3d782abb P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 1.
I20240502 07:58:14.753094 19000 leader_election.cc:302] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 345ab146afdf4d39a1d8583881c06743; no voters:
I20240502 07:58:14.753253 19489 raft_consensus.cc:2781] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:14.753350 19489 raft_consensus.cc:686] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 LEADER]: Becoming Leader. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Running, Role: LEADER
I20240502 07:58:14.753629 19489 consensus_queue.cc:237] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:14.756047 18526 catalog_manager.cc:5484] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132). New cstate: current_term: 1 leader_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:14.895046 19420 raft_consensus.cc:482] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:14.895210 19420 raft_consensus.cc:504] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.896296 19420 leader_election.cc:288] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523), d19f943e214647a3850f51a324017662 (127.18.13.133:40945)
I20240502 07:58:14.900789 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112385d70673444fb1c253255d47d8f8" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662" is_pre_election: true
I20240502 07:58:14.900789 18671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112385d70673444fb1c253255d47d8f8" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a" is_pre_election: true
I20240502 07:58:14.900947 19199 raft_consensus.cc:2454] T 112385d70673444fb1c253255d47d8f8 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:14.900947 18671 raft_consensus.cc:2454] T 112385d70673444fb1c253255d47d8f8 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:14.901309 18868 leader_election.cc:302] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f21202c69df478796a5d7dcfe8b07b5, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:14.901556 19420 raft_consensus.cc:2781] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:14.901619 19420 raft_consensus.cc:482] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:14.901647 19420 raft_consensus.cc:3037] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.902493 19420 raft_consensus.cc:504] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.902851 19420 leader_election.cc:288] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Requested vote from peers 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523), d19f943e214647a3850f51a324017662 (127.18.13.133:40945)
I20240502 07:58:14.903158 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112385d70673444fb1c253255d47d8f8" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662"
I20240502 07:58:14.903270 19199 raft_consensus.cc:3037] T 112385d70673444fb1c253255d47d8f8 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.903558 18671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112385d70673444fb1c253255d47d8f8" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a"
I20240502 07:58:14.903666 18671 raft_consensus.cc:3037] T 112385d70673444fb1c253255d47d8f8 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.904258 19199 raft_consensus.cc:2454] T 112385d70673444fb1c253255d47d8f8 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
I20240502 07:58:14.904526 18868 leader_election.cc:302] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f21202c69df478796a5d7dcfe8b07b5, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:14.904621 18671 raft_consensus.cc:2454] T 112385d70673444fb1c253255d47d8f8 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
I20240502 07:58:14.904677 19420 raft_consensus.cc:2781] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:14.904884 19420 raft_consensus.cc:686] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 LEADER]: Becoming Leader. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Running, Role: LEADER
I20240502 07:58:14.905184 19420 consensus_queue.cc:237] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:14.907522 18526 catalog_manager.cc:5484] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131). New cstate: current_term: 1 leader_uuid: "9f21202c69df478796a5d7dcfe8b07b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: HEALTHY } } }
W20240502 07:58:15.080832 19527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:15.081094 19527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:15.084350 19527 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 07:58:15.112982 19419 consensus_queue.cc:1035] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:15.115034 19419 consensus_queue.cc:1035] T f0b7aba425554d5f98024438383bbf37 P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:15.147819 19489 consensus_queue.cc:1035] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:15.151870 19489 consensus_queue.cc:1035] T fce88a610b8747d090d7fdda3d782abb P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:15.201674 18671 raft_consensus.cc:1261] T 112385d70673444fb1c253255d47d8f8 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Refusing update from remote peer 9f21202c69df478796a5d7dcfe8b07b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:15.201700 19199 raft_consensus.cc:1261] T 112385d70673444fb1c253255d47d8f8 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Refusing update from remote peer 9f21202c69df478796a5d7dcfe8b07b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:15.202157 19525 consensus_queue.cc:1035] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:15.202338 19420 consensus_queue.cc:1035] T 112385d70673444fb1c253255d47d8f8 P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 121.329 ms
time per row: 0.0606643 ms
I20240502 07:58:15.255080 18528 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45150:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:15.261006 18914 tablet_service.cc:1467] Processing CreateTablet for tablet b84243b4d86d49afa04c6abbc2a36bd1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=096bf1d16d324116ae5925b7aaa55b9a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.261343 18914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84243b4d86d49afa04c6abbc2a36bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.261510 19046 tablet_service.cc:1467] Processing CreateTablet for tablet cb668216cb3641f6a108a60b74717a70 (DEFAULT_TABLE table=rebalance_test_table_2 [id=096bf1d16d324116ae5925b7aaa55b9a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.261770 19046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb668216cb3641f6a108a60b74717a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.262349 19179 tablet_service.cc:1467] Processing CreateTablet for tablet b84243b4d86d49afa04c6abbc2a36bd1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=096bf1d16d324116ae5925b7aaa55b9a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.262653 19179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84243b4d86d49afa04c6abbc2a36bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.262591 18915 tablet_service.cc:1467] Processing CreateTablet for tablet 7fc479ebe934455aa25a07880c812a24 (DEFAULT_TABLE table=rebalance_test_table_2 [id=096bf1d16d324116ae5925b7aaa55b9a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.262830 18915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fc479ebe934455aa25a07880c812a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.264076 19178 tablet_service.cc:1467] Processing CreateTablet for tablet 7fc479ebe934455aa25a07880c812a24 (DEFAULT_TABLE table=rebalance_test_table_2 [id=096bf1d16d324116ae5925b7aaa55b9a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.264227 18650 tablet_service.cc:1467] Processing CreateTablet for tablet b84243b4d86d49afa04c6abbc2a36bd1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=096bf1d16d324116ae5925b7aaa55b9a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.264298 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fc479ebe934455aa25a07880c812a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.264377 18651 tablet_service.cc:1467] Processing CreateTablet for tablet 7fc479ebe934455aa25a07880c812a24 (DEFAULT_TABLE table=rebalance_test_table_2 [id=096bf1d16d324116ae5925b7aaa55b9a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.264477 18650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b84243b4d86d49afa04c6abbc2a36bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.264586 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fc479ebe934455aa25a07880c812a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.265672 19597 tablet_bootstrap.cc:492] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:15.265805 18782 tablet_service.cc:1467] Processing CreateTablet for tablet cb668216cb3641f6a108a60b74717a70 (DEFAULT_TABLE table=rebalance_test_table_2 [id=096bf1d16d324116ae5925b7aaa55b9a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.266065 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb668216cb3641f6a108a60b74717a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.266553 19310 tablet_service.cc:1467] Processing CreateTablet for tablet cb668216cb3641f6a108a60b74717a70 (DEFAULT_TABLE table=rebalance_test_table_2 [id=096bf1d16d324116ae5925b7aaa55b9a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.266855 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb668216cb3641f6a108a60b74717a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.267139 19597 tablet_bootstrap.cc:654] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.268359 19597 tablet_bootstrap.cc:492] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:15.268456 19597 ts_tablet_manager.cc:1367] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:15.268500 19598 tablet_bootstrap.cc:492] T b84243b4d86d49afa04c6abbc2a36bd1 P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:15.268997 19597 raft_consensus.cc:348] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:15.269130 19597 raft_consensus.cc:374] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.269167 19597 raft_consensus.cc:729] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.269308 19597 consensus_queue.cc:260] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:15.269410 19598 tablet_bootstrap.cc:654] T b84243b4d86d49afa04c6abbc2a36bd1 P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.269618 19597 ts_tablet_manager.cc:1398] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.269815 19597 tablet_bootstrap.cc:492] T 7fc479ebe934455aa25a07880c812a24 P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:15.270304 19595 tablet_bootstrap.cc:492] T b84243b4d86d49afa04c6abbc2a36bd1 P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:15.270922 19598 tablet_bootstrap.cc:492] T b84243b4d86d49afa04c6abbc2a36bd1 P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:15.271000 19598 ts_tablet_manager.cc:1367] T b84243b4d86d49afa04c6abbc2a36bd1 P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:58:15.271196 19597 tablet_bootstrap.cc:654] T 7fc479ebe934455aa25a07880c812a24 P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.271308 19595 tablet_bootstrap.cc:654] T b84243b4d86d49afa04c6abbc2a36bd1 P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.271466 19598 raft_consensus.cc:348] T b84243b4d86d49afa04c6abbc2a36bd1 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:15.271577 19598 raft_consensus.cc:374] T b84243b4d86d49afa04c6abbc2a36bd1 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.271596 19600 tablet_bootstrap.cc:492] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:15.271607 19598 raft_consensus.cc:729] T b84243b4d86d49afa04c6abbc2a36bd1 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.271759 19598 consensus_queue.cc:260] T b84243b4d86d49afa04c6abbc2a36bd1 P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:15.272068 19598 ts_tablet_manager.cc:1398] T b84243b4d86d49afa04c6abbc2a36bd1 P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.272228 19598 tablet_bootstrap.cc:492] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:15.272235 19601 tablet_bootstrap.cc:492] T cb668216cb3641f6a108a60b74717a70 P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:15.273054 19598 tablet_bootstrap.cc:654] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.273069 19601 tablet_bootstrap.cc:654] T cb668216cb3641f6a108a60b74717a70 P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.274466 19601 tablet_bootstrap.cc:492] T cb668216cb3641f6a108a60b74717a70 P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:15.274556 19601 ts_tablet_manager.cc:1367] T cb668216cb3641f6a108a60b74717a70 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:15.274644 19598 tablet_bootstrap.cc:492] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:15.274718 19598 ts_tablet_manager.cc:1367] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:58:15.275023 19601 raft_consensus.cc:348] T cb668216cb3641f6a108a60b74717a70 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:15.275125 19601 raft_consensus.cc:374] T cb668216cb3641f6a108a60b74717a70 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.275158 19601 raft_consensus.cc:729] T cb668216cb3641f6a108a60b74717a70 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.275118 19598 raft_consensus.cc:348] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:15.275210 19598 raft_consensus.cc:374] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.275244 19598 raft_consensus.cc:729] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.275288 19601 consensus_queue.cc:260] T cb668216cb3641f6a108a60b74717a70 P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:15.275346 19598 consensus_queue.cc:260] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:15.275576 19601 ts_tablet_manager.cc:1398] T cb668216cb3641f6a108a60b74717a70 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.275584 19598 ts_tablet_manager.cc:1398] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.275887 19597 tablet_bootstrap.cc:492] T 7fc479ebe934455aa25a07880c812a24 P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:15.275990 19597 ts_tablet_manager.cc:1367] T 7fc479ebe934455aa25a07880c812a24 P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:58:15.276463 19595 tablet_bootstrap.cc:492] T b84243b4d86d49afa04c6abbc2a36bd1 P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:15.276561 19595 ts_tablet_manager.cc:1367] T b84243b4d86d49afa04c6abbc2a36bd1 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:58:15.276513 19597 raft_consensus.cc:348] T 7fc479ebe934455aa25a07880c812a24 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:15.276618 19597 raft_consensus.cc:374] T 7fc479ebe934455aa25a07880c812a24 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.276674 19597 raft_consensus.cc:729] T 7fc479ebe934455aa25a07880c812a24 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.276803 19597 consensus_queue.cc:260] T 7fc479ebe934455aa25a07880c812a24 P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:15.277101 19597 ts_tablet_manager.cc:1398] T 7fc479ebe934455aa25a07880c812a24 P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.277099 19595 raft_consensus.cc:348] T b84243b4d86d49afa04c6abbc2a36bd1 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:15.277220 19595 raft_consensus.cc:374] T b84243b4d86d49afa04c6abbc2a36bd1 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.277257 19595 raft_consensus.cc:729] T b84243b4d86d49afa04c6abbc2a36bd1 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.277382 19595 consensus_queue.cc:260] T b84243b4d86d49afa04c6abbc2a36bd1 P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:15.277658 19595 ts_tablet_manager.cc:1398] T b84243b4d86d49afa04c6abbc2a36bd1 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.277838 19595 tablet_bootstrap.cc:492] T 7fc479ebe934455aa25a07880c812a24 P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:15.278807 19595 tablet_bootstrap.cc:654] T 7fc479ebe934455aa25a07880c812a24 P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.278848 19600 tablet_bootstrap.cc:654] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.279700 19596 tablet_bootstrap.cc:492] T cb668216cb3641f6a108a60b74717a70 P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:15.280212 19600 tablet_bootstrap.cc:492] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:15.280364 19595 tablet_bootstrap.cc:492] T 7fc479ebe934455aa25a07880c812a24 P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:15.280366 19600 ts_tablet_manager.cc:1367] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 07:58:15.280500 19595 ts_tablet_manager.cc:1367] T 7fc479ebe934455aa25a07880c812a24 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:15.280956 19595 raft_consensus.cc:348] T 7fc479ebe934455aa25a07880c812a24 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:15.281080 19595 raft_consensus.cc:374] T 7fc479ebe934455aa25a07880c812a24 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.281121 19595 raft_consensus.cc:729] T 7fc479ebe934455aa25a07880c812a24 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.281256 19595 consensus_queue.cc:260] T 7fc479ebe934455aa25a07880c812a24 P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:15.281939 19595 ts_tablet_manager.cc:1398] T 7fc479ebe934455aa25a07880c812a24 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.281975 19600 raft_consensus.cc:348] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:15.282120 19600 raft_consensus.cc:374] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.282171 19600 raft_consensus.cc:729] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.282303 19600 consensus_queue.cc:260] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:15.282652 19600 ts_tablet_manager.cc:1398] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:15.283059 19596 tablet_bootstrap.cc:654] T cb668216cb3641f6a108a60b74717a70 P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.284575 19596 tablet_bootstrap.cc:492] T cb668216cb3641f6a108a60b74717a70 P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:15.284651 19596 ts_tablet_manager.cc:1367] T cb668216cb3641f6a108a60b74717a70 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:58:15.285131 19596 raft_consensus.cc:348] T cb668216cb3641f6a108a60b74717a70 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:15.285252 19596 raft_consensus.cc:374] T cb668216cb3641f6a108a60b74717a70 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.285291 19596 raft_consensus.cc:729] T cb668216cb3641f6a108a60b74717a70 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.285426 19596 consensus_queue.cc:260] T cb668216cb3641f6a108a60b74717a70 P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:15.285759 19596 ts_tablet_manager.cc:1398] T cb668216cb3641f6a108a60b74717a70 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.297189 19584 raft_consensus.cc:482] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:15.297315 19584 raft_consensus.cc:504] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:15.298359 19584 leader_election.cc:288] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:15.302006 18934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84243b4d86d49afa04c6abbc2a36bd1" candidate_uuid: "d19f943e214647a3850f51a324017662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5" is_pre_election: true
I20240502 07:58:15.301981 18671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84243b4d86d49afa04c6abbc2a36bd1" candidate_uuid: "d19f943e214647a3850f51a324017662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a" is_pre_election: true
I20240502 07:58:15.302153 18934 raft_consensus.cc:2454] T b84243b4d86d49afa04c6abbc2a36bd1 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d19f943e214647a3850f51a324017662 in term 0.
I20240502 07:58:15.302153 18671 raft_consensus.cc:2454] T b84243b4d86d49afa04c6abbc2a36bd1 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d19f943e214647a3850f51a324017662 in term 0.
I20240502 07:58:15.302512 19134 leader_election.cc:302] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:15.302773 19584 raft_consensus.cc:2781] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:15.302842 19584 raft_consensus.cc:482] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:15.302891 19584 raft_consensus.cc:3037] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.303778 19584 raft_consensus.cc:504] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:15.304172 19584 leader_election.cc:288] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [CANDIDATE]: Term 1 election: Requested vote from peers 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:15.304481 18934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84243b4d86d49afa04c6abbc2a36bd1" candidate_uuid: "d19f943e214647a3850f51a324017662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5"
I20240502 07:58:15.304530 18671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b84243b4d86d49afa04c6abbc2a36bd1" candidate_uuid: "d19f943e214647a3850f51a324017662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a"
I20240502 07:58:15.304596 18934 raft_consensus.cc:3037] T b84243b4d86d49afa04c6abbc2a36bd1 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.304658 18671 raft_consensus.cc:3037] T b84243b4d86d49afa04c6abbc2a36bd1 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.305389 18934 raft_consensus.cc:2454] T b84243b4d86d49afa04c6abbc2a36bd1 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d19f943e214647a3850f51a324017662 in term 1.
I20240502 07:58:15.305402 18671 raft_consensus.cc:2454] T b84243b4d86d49afa04c6abbc2a36bd1 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d19f943e214647a3850f51a324017662 in term 1.
I20240502 07:58:15.305652 19134 leader_election.cc:302] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:15.305848 19584 raft_consensus.cc:2781] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:15.306053 19584 raft_consensus.cc:686] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [term 1 LEADER]: Becoming Leader. State: Replica: d19f943e214647a3850f51a324017662, State: Running, Role: LEADER
I20240502 07:58:15.306357 19584 consensus_queue.cc:237] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:15.308645 18527 catalog_manager.cc:5484] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 reported cstate change: term changed from 0 to 1, leader changed from <none> to d19f943e214647a3850f51a324017662 (127.18.13.133). New cstate: current_term: 1 leader_uuid: "d19f943e214647a3850f51a324017662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:15.349886 19558 raft_consensus.cc:482] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:15.350061 19558 raft_consensus.cc:504] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:15.350682 19558 leader_election.cc:288] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417), 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999)
I20240502 07:58:15.354171 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb668216cb3641f6a108a60b74717a70" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" is_pre_election: true
I20240502 07:58:15.354189 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb668216cb3641f6a108a60b74717a70" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260" is_pre_election: true
I20240502 07:58:15.354307 18803 raft_consensus.cc:2454] T cb668216cb3641f6a108a60b74717a70 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 0.
I20240502 07:58:15.354319 19067 raft_consensus.cc:2454] T cb668216cb3641f6a108a60b74717a70 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 0.
I20240502 07:58:15.354604 19264 leader_election.cc:302] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345ab146afdf4d39a1d8583881c06743, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:15.354768 19558 raft_consensus.cc:2781] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:15.354833 19558 raft_consensus.cc:482] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:15.354887 19558 raft_consensus.cc:3037] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.355669 19558 raft_consensus.cc:504] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:15.356025 19558 leader_election.cc:288] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 election: Requested vote from peers 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417), 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999)
I20240502 07:58:15.356352 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb668216cb3641f6a108a60b74717a70" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260"
I20240502 07:58:15.356321 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb668216cb3641f6a108a60b74717a70" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3"
I20240502 07:58:15.356452 18803 raft_consensus.cc:3037] T cb668216cb3641f6a108a60b74717a70 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.356524 19067 raft_consensus.cc:3037] T cb668216cb3641f6a108a60b74717a70 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.357190 18803 raft_consensus.cc:2454] T cb668216cb3641f6a108a60b74717a70 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 1.
I20240502 07:58:15.357281 19067 raft_consensus.cc:2454] T cb668216cb3641f6a108a60b74717a70 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 1.
I20240502 07:58:15.357453 19264 leader_election.cc:302] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345ab146afdf4d39a1d8583881c06743, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:15.357631 19558 raft_consensus.cc:2781] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:15.357743 19558 raft_consensus.cc:686] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [term 1 LEADER]: Becoming Leader. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Running, Role: LEADER
I20240502 07:58:15.357913 19558 consensus_queue.cc:237] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:15.359640 18527 catalog_manager.cc:5484] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 reported cstate change: term changed from 0 to 1, leader changed from <none> to 345ab146afdf4d39a1d8583881c06743 (127.18.13.134). New cstate: current_term: 1 leader_uuid: "345ab146afdf4d39a1d8583881c06743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:15.484514 19425 raft_consensus.cc:482] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:15.484669 19425 raft_consensus.cc:504] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:15.485116 19425 leader_election.cc:288] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389)
I20240502 07:58:15.485569 18934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc479ebe934455aa25a07880c812a24" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5" is_pre_election: true
I20240502 07:58:15.485591 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc479ebe934455aa25a07880c812a24" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662" is_pre_election: true
I20240502 07:58:15.485689 18934 raft_consensus.cc:2454] T 7fc479ebe934455aa25a07880c812a24 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 0.
I20240502 07:58:15.485749 19199 raft_consensus.cc:2454] T 7fc479ebe934455aa25a07880c812a24 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 0.
I20240502 07:58:15.485998 18605 leader_election.cc:302] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, 9f21202c69df478796a5d7dcfe8b07b5; no voters:
I20240502 07:58:15.486171 19425 raft_consensus.cc:2781] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:15.486224 19425 raft_consensus.cc:482] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:15.486250 19425 raft_consensus.cc:3037] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.487104 19425 raft_consensus.cc:504] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:15.487401 19425 leader_election.cc:288] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 election: Requested vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389)
I20240502 07:58:15.487704 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc479ebe934455aa25a07880c812a24" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662"
I20240502 07:58:15.487725 18934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fc479ebe934455aa25a07880c812a24" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5"
I20240502 07:58:15.487840 19199 raft_consensus.cc:3037] T 7fc479ebe934455aa25a07880c812a24 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.487856 18934 raft_consensus.cc:3037] T 7fc479ebe934455aa25a07880c812a24 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.488730 19199 raft_consensus.cc:2454] T 7fc479ebe934455aa25a07880c812a24 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 1.
I20240502 07:58:15.488729 18934 raft_consensus.cc:2454] T 7fc479ebe934455aa25a07880c812a24 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 1.
I20240502 07:58:15.489039 18604 leader_election.cc:302] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:15.489181 19425 raft_consensus.cc:2781] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:15.489509 19425 raft_consensus.cc:686] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [term 1 LEADER]: Becoming Leader. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Running, Role: LEADER
I20240502 07:58:15.489681 19425 consensus_queue.cc:237] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:15.491423 18527 catalog_manager.cc:5484] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f6bc76c600c413d9f72526563d3234a (127.18.13.129). New cstate: current_term: 1 leader_uuid: "2f6bc76c600c413d9f72526563d3234a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:15.614362 19619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:15.614588 19619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:15.617689 19619 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 07:58:15.708771 19067 raft_consensus.cc:1261] T cb668216cb3641f6a108a60b74717a70 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Refusing update from remote peer 345ab146afdf4d39a1d8583881c06743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:15.709236 18803 raft_consensus.cc:1261] T cb668216cb3641f6a108a60b74717a70 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Refusing update from remote peer 345ab146afdf4d39a1d8583881c06743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:15.709249 19429 consensus_queue.cc:1035] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:15.709833 19429 consensus_queue.cc:1035] T cb668216cb3641f6a108a60b74717a70 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:15.714242 18671 raft_consensus.cc:1261] T b84243b4d86d49afa04c6abbc2a36bd1 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Refusing update from remote peer d19f943e214647a3850f51a324017662: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:15.714947 18934 raft_consensus.cc:1261] T b84243b4d86d49afa04c6abbc2a36bd1 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Refusing update from remote peer d19f943e214647a3850f51a324017662: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:15.716367 19584 consensus_queue.cc:1035] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:15.716817 19584 consensus_queue.cc:1035] T b84243b4d86d49afa04c6abbc2a36bd1 P d19f943e214647a3850f51a324017662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:15.717743 18934 raft_consensus.cc:1261] T 7fc479ebe934455aa25a07880c812a24 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Refusing update from remote peer 2f6bc76c600c413d9f72526563d3234a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:15.718212 19617 consensus_queue.cc:1035] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:15.720573 19199 raft_consensus.cc:1261] T 7fc479ebe934455aa25a07880c812a24 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Refusing update from remote peer 2f6bc76c600c413d9f72526563d3234a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:15.721221 19425 consensus_queue.cc:1035] T 7fc479ebe934455aa25a07880c812a24 P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:15.754355 19652 mvcc.cc:205] Tried to move back new op lower bound from 7023151905618313216 to 7023151904187387904. Current Snapshot: MvccSnapshot[applied={T|T < 7023151905618313216}]
INSERT report
rows total: 2000
time total: 126.281 ms
time per row: 0.0631406 ms
I20240502 07:58:15.786180 18524 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45150:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:15.792330 19046 tablet_service.cc:1467] Processing CreateTablet for tablet a0ae282afe034174bb547fe021b725ae (DEFAULT_TABLE table=rebalance_test_table_3 [id=3b79519ca6774095b4bec77ce2b010ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.792487 18915 tablet_service.cc:1467] Processing CreateTablet for tablet 062558123ed44b04969b84580b8678c9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3b79519ca6774095b4bec77ce2b010ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.792472 19047 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc3357e14f24f0682bebbbba109d401 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3b79519ca6774095b4bec77ce2b010ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.792562 18782 tablet_service.cc:1467] Processing CreateTablet for tablet a0ae282afe034174bb547fe021b725ae (DEFAULT_TABLE table=rebalance_test_table_3 [id=3b79519ca6774095b4bec77ce2b010ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.792654 19046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0ae282afe034174bb547fe021b725ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.792742 18915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 062558123ed44b04969b84580b8678c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.792810 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0ae282afe034174bb547fe021b725ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.792892 19047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc3357e14f24f0682bebbbba109d401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.795161 19669 tablet_bootstrap.cc:492] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:15.795200 19670 tablet_bootstrap.cc:492] T 062558123ed44b04969b84580b8678c9 P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:15.795233 18651 tablet_service.cc:1467] Processing CreateTablet for tablet 062558123ed44b04969b84580b8678c9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3b79519ca6774095b4bec77ce2b010ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.795521 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 062558123ed44b04969b84580b8678c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.796133 19669 tablet_bootstrap.cc:654] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.796133 19670 tablet_bootstrap.cc:654] T 062558123ed44b04969b84580b8678c9 P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.797164 19669 tablet_bootstrap.cc:492] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:15.797240 19669 ts_tablet_manager.cc:1367] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:15.797441 19670 tablet_bootstrap.cc:492] T 062558123ed44b04969b84580b8678c9 P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:15.797520 19670 ts_tablet_manager.cc:1367] T 062558123ed44b04969b84580b8678c9 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:15.797868 19669 raft_consensus.cc:348] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:15.797995 19669 raft_consensus.cc:374] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.797961 19670 raft_consensus.cc:348] T 062558123ed44b04969b84580b8678c9 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:15.798028 19669 raft_consensus.cc:729] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.798053 19670 raft_consensus.cc:374] T 062558123ed44b04969b84580b8678c9 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.798086 19670 raft_consensus.cc:729] T 062558123ed44b04969b84580b8678c9 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.798146 19669 consensus_queue.cc:260] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:15.798218 19670 consensus_queue.cc:260] T 062558123ed44b04969b84580b8678c9 P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:15.798260 19310 tablet_service.cc:1467] Processing CreateTablet for tablet a0ae282afe034174bb547fe021b725ae (DEFAULT_TABLE table=rebalance_test_table_3 [id=3b79519ca6774095b4bec77ce2b010ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.798453 19311 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc3357e14f24f0682bebbbba109d401 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3b79519ca6774095b4bec77ce2b010ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.798506 19669 ts_tablet_manager.cc:1398] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.798528 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0ae282afe034174bb547fe021b725ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.798532 19670 ts_tablet_manager.cc:1398] T 062558123ed44b04969b84580b8678c9 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.798645 19311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc3357e14f24f0682bebbbba109d401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.798656 19669 tablet_bootstrap.cc:492] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:15.799288 19669 tablet_bootstrap.cc:654] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.799894 19674 tablet_bootstrap.cc:492] T 062558123ed44b04969b84580b8678c9 P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:15.800665 19674 tablet_bootstrap.cc:654] T 062558123ed44b04969b84580b8678c9 P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.800935 19178 tablet_service.cc:1467] Processing CreateTablet for tablet 062558123ed44b04969b84580b8678c9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3b79519ca6774095b4bec77ce2b010ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.801116 19669 tablet_bootstrap.cc:492] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:15.801190 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 062558123ed44b04969b84580b8678c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.801198 19669 ts_tablet_manager.cc:1367] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:58:15.801815 19669 raft_consensus.cc:348] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:15.801905 19669 raft_consensus.cc:374] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.801940 19669 raft_consensus.cc:729] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.802107 19669 consensus_queue.cc:260] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:15.802353 19669 ts_tablet_manager.cc:1398] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:15.803062 19678 tablet_bootstrap.cc:492] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:15.803470 19674 tablet_bootstrap.cc:492] T 062558123ed44b04969b84580b8678c9 P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:15.803594 19674 ts_tablet_manager.cc:1367] T 062558123ed44b04969b84580b8678c9 P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:15.804023 19678 tablet_bootstrap.cc:654] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.804153 19674 raft_consensus.cc:348] T 062558123ed44b04969b84580b8678c9 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:15.804316 19674 raft_consensus.cc:374] T 062558123ed44b04969b84580b8678c9 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.804396 19674 raft_consensus.cc:729] T 062558123ed44b04969b84580b8678c9 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.804600 19674 consensus_queue.cc:260] T 062558123ed44b04969b84580b8678c9 P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:15.804986 19674 ts_tablet_manager.cc:1398] T 062558123ed44b04969b84580b8678c9 P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.805090 18782 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc3357e14f24f0682bebbbba109d401 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3b79519ca6774095b4bec77ce2b010ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.805384 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc3357e14f24f0682bebbbba109d401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.807637 19676 tablet_bootstrap.cc:492] T 8bc3357e14f24f0682bebbbba109d401 P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:15.807974 19671 tablet_bootstrap.cc:492] T a0ae282afe034174bb547fe021b725ae P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:15.808557 19676 tablet_bootstrap.cc:654] T 8bc3357e14f24f0682bebbbba109d401 P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.808703 19671 tablet_bootstrap.cc:654] T a0ae282afe034174bb547fe021b725ae P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.809733 19678 tablet_bootstrap.cc:492] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:15.809733 19671 tablet_bootstrap.cc:492] T a0ae282afe034174bb547fe021b725ae P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:15.809828 19671 ts_tablet_manager.cc:1367] T a0ae282afe034174bb547fe021b725ae P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:15.809828 19678 ts_tablet_manager.cc:1367] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 07:58:15.809900 19676 tablet_bootstrap.cc:492] T 8bc3357e14f24f0682bebbbba109d401 P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:15.809974 19676 ts_tablet_manager.cc:1367] T 8bc3357e14f24f0682bebbbba109d401 P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:15.810256 19671 raft_consensus.cc:348] T a0ae282afe034174bb547fe021b725ae P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:15.810256 19678 raft_consensus.cc:348] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:15.810354 19671 raft_consensus.cc:374] T a0ae282afe034174bb547fe021b725ae P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.810354 19678 raft_consensus.cc:374] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.810386 19678 raft_consensus.cc:729] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.810386 19671 raft_consensus.cc:729] T a0ae282afe034174bb547fe021b725ae P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.810360 19676 raft_consensus.cc:348] T 8bc3357e14f24f0682bebbbba109d401 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:15.810461 19676 raft_consensus.cc:374] T 8bc3357e14f24f0682bebbbba109d401 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.810494 19676 raft_consensus.cc:729] T 8bc3357e14f24f0682bebbbba109d401 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.810551 19671 consensus_queue.cc:260] T a0ae282afe034174bb547fe021b725ae P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:15.810626 19676 consensus_queue.cc:260] T 8bc3357e14f24f0682bebbbba109d401 P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:15.810891 19676 ts_tablet_manager.cc:1398] T 8bc3357e14f24f0682bebbbba109d401 P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.811116 19676 tablet_bootstrap.cc:492] T a0ae282afe034174bb547fe021b725ae P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:15.811388 19678 consensus_queue.cc:260] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:15.811959 19678 ts_tablet_manager.cc:1398] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:15.812074 19676 tablet_bootstrap.cc:654] T a0ae282afe034174bb547fe021b725ae P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.812500 19671 ts_tablet_manager.cc:1398] T a0ae282afe034174bb547fe021b725ae P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:58:15.812922 19671 tablet_bootstrap.cc:492] T 8bc3357e14f24f0682bebbbba109d401 P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:15.813520 19676 tablet_bootstrap.cc:492] T a0ae282afe034174bb547fe021b725ae P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:15.813613 19676 ts_tablet_manager.cc:1367] T a0ae282afe034174bb547fe021b725ae P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:15.814030 19671 tablet_bootstrap.cc:654] T 8bc3357e14f24f0682bebbbba109d401 P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.814102 19676 raft_consensus.cc:348] T a0ae282afe034174bb547fe021b725ae P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:15.814224 19676 raft_consensus.cc:374] T a0ae282afe034174bb547fe021b725ae P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.814270 19676 raft_consensus.cc:729] T a0ae282afe034174bb547fe021b725ae P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.814400 19676 consensus_queue.cc:260] T a0ae282afe034174bb547fe021b725ae P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:15.814667 19676 ts_tablet_manager.cc:1398] T a0ae282afe034174bb547fe021b725ae P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.815634 19671 tablet_bootstrap.cc:492] T 8bc3357e14f24f0682bebbbba109d401 P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:15.815721 19671 ts_tablet_manager.cc:1367] T 8bc3357e14f24f0682bebbbba109d401 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:15.816195 19671 raft_consensus.cc:348] T 8bc3357e14f24f0682bebbbba109d401 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:15.816296 19671 raft_consensus.cc:374] T 8bc3357e14f24f0682bebbbba109d401 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.816339 19671 raft_consensus.cc:729] T 8bc3357e14f24f0682bebbbba109d401 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.816480 19671 consensus_queue.cc:260] T 8bc3357e14f24f0682bebbbba109d401 P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:15.816710 19671 ts_tablet_manager.cc:1398] T 8bc3357e14f24f0682bebbbba109d401 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.827462 19584 raft_consensus.cc:482] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:15.827584 19584 raft_consensus.cc:504] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:15.828013 19584 leader_election.cc:288] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:15.828366 18935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062558123ed44b04969b84580b8678c9" candidate_uuid: "d19f943e214647a3850f51a324017662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5" is_pre_election: true
I20240502 07:58:15.828372 18671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062558123ed44b04969b84580b8678c9" candidate_uuid: "d19f943e214647a3850f51a324017662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a" is_pre_election: true
I20240502 07:58:15.828501 18935 raft_consensus.cc:2454] T 062558123ed44b04969b84580b8678c9 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d19f943e214647a3850f51a324017662 in term 0.
I20240502 07:58:15.828510 18671 raft_consensus.cc:2454] T 062558123ed44b04969b84580b8678c9 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d19f943e214647a3850f51a324017662 in term 0.
I20240502 07:58:15.828770 19133 leader_election.cc:302] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f21202c69df478796a5d7dcfe8b07b5, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:15.828940 19584 raft_consensus.cc:2781] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:15.829003 19584 raft_consensus.cc:482] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:15.829046 19584 raft_consensus.cc:3037] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.830018 19584 raft_consensus.cc:504] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:15.830336 19584 leader_election.cc:288] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [CANDIDATE]: Term 1 election: Requested vote from peers 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:15.830621 18935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062558123ed44b04969b84580b8678c9" candidate_uuid: "d19f943e214647a3850f51a324017662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5"
I20240502 07:58:15.830718 18935 raft_consensus.cc:3037] T 062558123ed44b04969b84580b8678c9 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.830689 18671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062558123ed44b04969b84580b8678c9" candidate_uuid: "d19f943e214647a3850f51a324017662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a"
I20240502 07:58:15.830794 18671 raft_consensus.cc:3037] T 062558123ed44b04969b84580b8678c9 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.831523 18671 raft_consensus.cc:2454] T 062558123ed44b04969b84580b8678c9 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d19f943e214647a3850f51a324017662 in term 1.
I20240502 07:58:15.831594 18935 raft_consensus.cc:2454] T 062558123ed44b04969b84580b8678c9 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d19f943e214647a3850f51a324017662 in term 1.
I20240502 07:58:15.831758 19134 leader_election.cc:302] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:15.831974 19584 raft_consensus.cc:2781] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:15.832046 19584 raft_consensus.cc:686] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [term 1 LEADER]: Becoming Leader. State: Replica: d19f943e214647a3850f51a324017662, State: Running, Role: LEADER
I20240502 07:58:15.832181 19584 consensus_queue.cc:237] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:15.833654 18525 catalog_manager.cc:5484] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 reported cstate change: term changed from 0 to 1, leader changed from <none> to d19f943e214647a3850f51a324017662 (127.18.13.133). New cstate: current_term: 1 leader_uuid: "d19f943e214647a3850f51a324017662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:15.955247 19489 raft_consensus.cc:482] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:15.955363 19489 raft_consensus.cc:504] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:15.955739 19489 leader_election.cc:288] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999), 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647)
I20240502 07:58:15.956166 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ae282afe034174bb547fe021b725ae" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260" is_pre_election: true
I20240502 07:58:15.956199 19330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ae282afe034174bb547fe021b725ae" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743" is_pre_election: true
I20240502 07:58:15.956317 19330 raft_consensus.cc:2454] T a0ae282afe034174bb547fe021b725ae P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 0.
I20240502 07:58:15.956317 18803 raft_consensus.cc:2454] T a0ae282afe034174bb547fe021b725ae P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 0.
I20240502 07:58:15.956653 19000 leader_election.cc:302] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 345ab146afdf4d39a1d8583881c06743; no voters:
I20240502 07:58:15.956821 19489 raft_consensus.cc:2781] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:15.956871 19489 raft_consensus.cc:482] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:15.956907 19489 raft_consensus.cc:3037] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.957690 19489 raft_consensus.cc:504] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:15.957988 19489 leader_election.cc:288] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 election: Requested vote from peers 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999), 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647)
I20240502 07:58:15.958304 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ae282afe034174bb547fe021b725ae" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260"
I20240502 07:58:15.958339 19330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0ae282afe034174bb547fe021b725ae" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743"
I20240502 07:58:15.958421 18803 raft_consensus.cc:3037] T a0ae282afe034174bb547fe021b725ae P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.958436 19330 raft_consensus.cc:3037] T a0ae282afe034174bb547fe021b725ae P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.959127 19330 raft_consensus.cc:2454] T a0ae282afe034174bb547fe021b725ae P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 1.
I20240502 07:58:15.959218 18803 raft_consensus.cc:2454] T a0ae282afe034174bb547fe021b725ae P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 1.
I20240502 07:58:15.959398 19000 leader_election.cc:302] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 345ab146afdf4d39a1d8583881c06743; no voters:
I20240502 07:58:15.959568 19489 raft_consensus.cc:2781] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:15.959717 19489 raft_consensus.cc:686] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 LEADER]: Becoming Leader. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Running, Role: LEADER
I20240502 07:58:15.959890 19489 consensus_queue.cc:237] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:15.961455 18525 catalog_manager.cc:5484] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132). New cstate: current_term: 1 leader_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:16.041052 19489 raft_consensus.cc:482] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:16.041183 19489 raft_consensus.cc:504] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:16.041617 19489 leader_election.cc:288] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647), 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999)
I20240502 07:58:16.042044 19330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc3357e14f24f0682bebbbba109d401" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743" is_pre_election: true
I20240502 07:58:16.042068 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc3357e14f24f0682bebbbba109d401" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260" is_pre_election: true
I20240502 07:58:16.042186 19330 raft_consensus.cc:2454] T 8bc3357e14f24f0682bebbbba109d401 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 0.
I20240502 07:58:16.042223 18803 raft_consensus.cc:2454] T 8bc3357e14f24f0682bebbbba109d401 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 0.
I20240502 07:58:16.042517 19000 leader_election.cc:302] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:16.042670 19489 raft_consensus.cc:2781] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:16.042724 19489 raft_consensus.cc:482] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:16.042770 19489 raft_consensus.cc:3037] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.043543 19489 raft_consensus.cc:504] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:16.043877 19489 leader_election.cc:288] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 election: Requested vote from peers 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647), 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999)
I20240502 07:58:16.044140 19330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc3357e14f24f0682bebbbba109d401" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743"
I20240502 07:58:16.044250 19330 raft_consensus.cc:3037] T 8bc3357e14f24f0682bebbbba109d401 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.044273 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc3357e14f24f0682bebbbba109d401" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260"
I20240502 07:58:16.044392 18803 raft_consensus.cc:3037] T 8bc3357e14f24f0682bebbbba109d401 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.045090 18803 raft_consensus.cc:2454] T 8bc3357e14f24f0682bebbbba109d401 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 1.
I20240502 07:58:16.045163 19330 raft_consensus.cc:2454] T 8bc3357e14f24f0682bebbbba109d401 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 1.
I20240502 07:58:16.045321 19000 leader_election.cc:302] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:16.045467 19489 raft_consensus.cc:2781] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:16.045552 19489 raft_consensus.cc:686] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 LEADER]: Becoming Leader. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Running, Role: LEADER
I20240502 07:58:16.045676 19489 consensus_queue.cc:237] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:16.047384 18525 catalog_manager.cc:5484] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132). New cstate: current_term: 1 leader_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: HEALTHY } } }
W20240502 07:58:16.204919 19688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:16.205214 19688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:16.208683 19688 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 07:58:16.231353 19584 consensus_queue.cc:1035] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.235299 19584 consensus_queue.cc:1035] T 062558123ed44b04969b84580b8678c9 P d19f943e214647a3850f51a324017662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.291200 19330 raft_consensus.cc:1261] T a0ae282afe034174bb547fe021b725ae P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Refusing update from remote peer 341d0eb74c3544faaa2633d7aa3f77b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.291638 18803 raft_consensus.cc:1261] T a0ae282afe034174bb547fe021b725ae P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Refusing update from remote peer 341d0eb74c3544faaa2633d7aa3f77b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.296972 19685 consensus_queue.cc:1035] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.297353 19685 consensus_queue.cc:1035] T a0ae282afe034174bb547fe021b725ae P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.297518 18803 raft_consensus.cc:1261] T 8bc3357e14f24f0682bebbbba109d401 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Refusing update from remote peer 341d0eb74c3544faaa2633d7aa3f77b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.297612 19330 raft_consensus.cc:1261] T 8bc3357e14f24f0682bebbbba109d401 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Refusing update from remote peer 341d0eb74c3544faaa2633d7aa3f77b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.298020 19489 consensus_queue.cc:1035] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.300127 19489 consensus_queue.cc:1035] T 8bc3357e14f24f0682bebbbba109d401 P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.506 ms
time per row: 0.0527529 ms
I20240502 07:58:16.358855 18521 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45150:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:16.364101 18915 tablet_service.cc:1467] Processing CreateTablet for tablet 822ecfcf16fa4c60897cc18d14138289 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7967eab377664818946aeb59515bd22d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.364169 19178 tablet_service.cc:1467] Processing CreateTablet for tablet 822ecfcf16fa4c60897cc18d14138289 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7967eab377664818946aeb59515bd22d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.364413 18915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822ecfcf16fa4c60897cc18d14138289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.364413 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822ecfcf16fa4c60897cc18d14138289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.364820 18650 tablet_service.cc:1467] Processing CreateTablet for tablet 46247e07dbff4195b8681adb87bccbce (DEFAULT_TABLE table=rebalance_test_table_4 [id=7967eab377664818946aeb59515bd22d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.365132 18650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46247e07dbff4195b8681adb87bccbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.365614 19310 tablet_service.cc:1467] Processing CreateTablet for tablet 96cddef766ee4366991da031cbc8e2d5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7967eab377664818946aeb59515bd22d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.365856 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96cddef766ee4366991da031cbc8e2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.366854 19046 tablet_service.cc:1467] Processing CreateTablet for tablet 96cddef766ee4366991da031cbc8e2d5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7967eab377664818946aeb59515bd22d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.366922 19178 tablet_service.cc:1467] Processing CreateTablet for tablet 46247e07dbff4195b8681adb87bccbce (DEFAULT_TABLE table=rebalance_test_table_4 [id=7967eab377664818946aeb59515bd22d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.367112 19046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96cddef766ee4366991da031cbc8e2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.367112 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46247e07dbff4195b8681adb87bccbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.364820 18651 tablet_service.cc:1467] Processing CreateTablet for tablet 822ecfcf16fa4c60897cc18d14138289 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7967eab377664818946aeb59515bd22d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.367350 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 822ecfcf16fa4c60897cc18d14138289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.368278 18782 tablet_service.cc:1467] Processing CreateTablet for tablet 96cddef766ee4366991da031cbc8e2d5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=7967eab377664818946aeb59515bd22d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.368551 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96cddef766ee4366991da031cbc8e2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.369203 19750 tablet_bootstrap.cc:492] T 46247e07dbff4195b8681adb87bccbce P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:16.369845 19749 tablet_bootstrap.cc:492] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:16.370203 19750 tablet_bootstrap.cc:654] T 46247e07dbff4195b8681adb87bccbce P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.370774 19749 tablet_bootstrap.cc:654] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.371372 19751 tablet_bootstrap.cc:492] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:16.371430 18915 tablet_service.cc:1467] Processing CreateTablet for tablet 46247e07dbff4195b8681adb87bccbce (DEFAULT_TABLE table=rebalance_test_table_4 [id=7967eab377664818946aeb59515bd22d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.371657 18915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46247e07dbff4195b8681adb87bccbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.371954 19752 tablet_bootstrap.cc:492] T 96cddef766ee4366991da031cbc8e2d5 P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:16.372299 19751 tablet_bootstrap.cc:654] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.372835 19752 tablet_bootstrap.cc:654] T 96cddef766ee4366991da031cbc8e2d5 P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.372990 19753 tablet_bootstrap.cc:492] T 96cddef766ee4366991da031cbc8e2d5 P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:16.373399 19751 tablet_bootstrap.cc:492] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:16.373474 19749 tablet_bootstrap.cc:492] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:16.373536 19751 ts_tablet_manager.cc:1367] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:16.373579 19749 ts_tablet_manager.cc:1367] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:16.373828 19752 tablet_bootstrap.cc:492] T 96cddef766ee4366991da031cbc8e2d5 P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:16.373919 19752 ts_tablet_manager.cc:1367] T 96cddef766ee4366991da031cbc8e2d5 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:16.374008 19751 raft_consensus.cc:348] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:16.374153 19751 raft_consensus.cc:374] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.374218 19751 raft_consensus.cc:729] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.374114 19749 raft_consensus.cc:348] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:16.374271 19753 tablet_bootstrap.cc:654] T 96cddef766ee4366991da031cbc8e2d5 P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.374353 19749 raft_consensus.cc:374] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.374378 19751 consensus_queue.cc:260] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:16.374444 19749 raft_consensus.cc:729] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.374419 19752 raft_consensus.cc:348] T 96cddef766ee4366991da031cbc8e2d5 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:16.374526 19752 raft_consensus.cc:374] T 96cddef766ee4366991da031cbc8e2d5 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.374572 19752 raft_consensus.cc:729] T 96cddef766ee4366991da031cbc8e2d5 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.374648 19749 consensus_queue.cc:260] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:16.374720 19751 ts_tablet_manager.cc:1398] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.374693 19752 consensus_queue.cc:260] T 96cddef766ee4366991da031cbc8e2d5 P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:16.374964 19752 ts_tablet_manager.cc:1398] T 96cddef766ee4366991da031cbc8e2d5 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.375452 19750 tablet_bootstrap.cc:492] T 46247e07dbff4195b8681adb87bccbce P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:16.375541 19750 ts_tablet_manager.cc:1367] T 46247e07dbff4195b8681adb87bccbce P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 07:58:16.375546 19749 ts_tablet_manager.cc:1398] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:16.375705 19749 tablet_bootstrap.cc:492] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:16.375986 19750 raft_consensus.cc:348] T 46247e07dbff4195b8681adb87bccbce P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:16.376093 19750 raft_consensus.cc:374] T 46247e07dbff4195b8681adb87bccbce P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.376129 19750 raft_consensus.cc:729] T 46247e07dbff4195b8681adb87bccbce P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.376240 19750 consensus_queue.cc:260] T 46247e07dbff4195b8681adb87bccbce P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:16.376544 19750 ts_tablet_manager.cc:1398] T 46247e07dbff4195b8681adb87bccbce P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.376731 19750 tablet_bootstrap.cc:492] T 822ecfcf16fa4c60897cc18d14138289 P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:16.377512 19750 tablet_bootstrap.cc:654] T 822ecfcf16fa4c60897cc18d14138289 P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.378861 19753 tablet_bootstrap.cc:492] T 96cddef766ee4366991da031cbc8e2d5 P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:16.378957 19753 ts_tablet_manager.cc:1367] T 96cddef766ee4366991da031cbc8e2d5 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:58:16.379328 19753 raft_consensus.cc:348] T 96cddef766ee4366991da031cbc8e2d5 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:16.379428 19753 raft_consensus.cc:374] T 96cddef766ee4366991da031cbc8e2d5 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.379465 19753 raft_consensus.cc:729] T 96cddef766ee4366991da031cbc8e2d5 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.379711 19749 tablet_bootstrap.cc:654] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.379648 19753 consensus_queue.cc:260] T 96cddef766ee4366991da031cbc8e2d5 P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:16.379918 19748 tablet_bootstrap.cc:492] T 822ecfcf16fa4c60897cc18d14138289 P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:16.379951 19753 ts_tablet_manager.cc:1398] T 96cddef766ee4366991da031cbc8e2d5 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:16.380640 19750 tablet_bootstrap.cc:492] T 822ecfcf16fa4c60897cc18d14138289 P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:16.380721 19750 ts_tablet_manager.cc:1367] T 822ecfcf16fa4c60897cc18d14138289 P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:16.381206 19748 tablet_bootstrap.cc:654] T 822ecfcf16fa4c60897cc18d14138289 P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.381194 19750 raft_consensus.cc:348] T 822ecfcf16fa4c60897cc18d14138289 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:16.381292 19750 raft_consensus.cc:374] T 822ecfcf16fa4c60897cc18d14138289 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.381325 19750 raft_consensus.cc:729] T 822ecfcf16fa4c60897cc18d14138289 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.381407 19749 tablet_bootstrap.cc:492] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:16.381431 19750 consensus_queue.cc:260] T 822ecfcf16fa4c60897cc18d14138289 P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:16.381500 19749 ts_tablet_manager.cc:1367] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240502 07:58:16.381700 19750 ts_tablet_manager.cc:1398] T 822ecfcf16fa4c60897cc18d14138289 P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.381953 19749 raft_consensus.cc:348] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:16.382059 19749 raft_consensus.cc:374] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.382092 19749 raft_consensus.cc:729] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.382211 19749 consensus_queue.cc:260] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:16.382491 19749 ts_tablet_manager.cc:1398] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:16.383404 19748 tablet_bootstrap.cc:492] T 822ecfcf16fa4c60897cc18d14138289 P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:16.383494 19748 ts_tablet_manager.cc:1367] T 822ecfcf16fa4c60897cc18d14138289 P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:58:16.383961 19748 raft_consensus.cc:348] T 822ecfcf16fa4c60897cc18d14138289 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:16.384075 19748 raft_consensus.cc:374] T 822ecfcf16fa4c60897cc18d14138289 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.384109 19748 raft_consensus.cc:729] T 822ecfcf16fa4c60897cc18d14138289 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.384236 19748 consensus_queue.cc:260] T 822ecfcf16fa4c60897cc18d14138289 P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:16.384474 19748 ts_tablet_manager.cc:1398] T 822ecfcf16fa4c60897cc18d14138289 P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.384632 19748 tablet_bootstrap.cc:492] T 46247e07dbff4195b8681adb87bccbce P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:16.385452 19748 tablet_bootstrap.cc:654] T 46247e07dbff4195b8681adb87bccbce P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.386495 19748 tablet_bootstrap.cc:492] T 46247e07dbff4195b8681adb87bccbce P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:16.386582 19748 ts_tablet_manager.cc:1367] T 46247e07dbff4195b8681adb87bccbce P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:16.386996 19748 raft_consensus.cc:348] T 46247e07dbff4195b8681adb87bccbce P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:16.387112 19748 raft_consensus.cc:374] T 46247e07dbff4195b8681adb87bccbce P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.387145 19748 raft_consensus.cc:729] T 46247e07dbff4195b8681adb87bccbce P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.387233 19748 consensus_queue.cc:260] T 46247e07dbff4195b8681adb87bccbce P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:16.387488 19748 ts_tablet_manager.cc:1398] T 46247e07dbff4195b8681adb87bccbce P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.441725 19420 raft_consensus.cc:482] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:16.441864 19420 raft_consensus.cc:504] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:16.442301 19420 leader_election.cc:288] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:16.442788 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46247e07dbff4195b8681adb87bccbce" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662" is_pre_election: true
I20240502 07:58:16.442813 18671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46247e07dbff4195b8681adb87bccbce" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a" is_pre_election: true
I20240502 07:58:16.442937 19199 raft_consensus.cc:2454] T 46247e07dbff4195b8681adb87bccbce P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:16.442960 18671 raft_consensus.cc:2454] T 46247e07dbff4195b8681adb87bccbce P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:16.443214 18868 leader_election.cc:302] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f21202c69df478796a5d7dcfe8b07b5, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:16.443393 19420 raft_consensus.cc:2781] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:16.443472 19420 raft_consensus.cc:482] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:16.443522 19420 raft_consensus.cc:3037] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.444403 19420 raft_consensus.cc:504] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:16.444744 19420 leader_election.cc:288] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Requested vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:16.445020 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46247e07dbff4195b8681adb87bccbce" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662"
I20240502 07:58:16.445123 19198 raft_consensus.cc:3037] T 46247e07dbff4195b8681adb87bccbce P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.445119 18671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46247e07dbff4195b8681adb87bccbce" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a"
I20240502 07:58:16.445238 18671 raft_consensus.cc:3037] T 46247e07dbff4195b8681adb87bccbce P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.445878 19198 raft_consensus.cc:2454] T 46247e07dbff4195b8681adb87bccbce P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
I20240502 07:58:16.446002 18671 raft_consensus.cc:2454] T 46247e07dbff4195b8681adb87bccbce P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
I20240502 07:58:16.446188 18868 leader_election.cc:302] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f21202c69df478796a5d7dcfe8b07b5, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:16.446377 19420 raft_consensus.cc:2781] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:16.446566 19420 raft_consensus.cc:686] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 LEADER]: Becoming Leader. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Running, Role: LEADER
I20240502 07:58:16.446712 19420 consensus_queue.cc:237] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:16.448249 18527 catalog_manager.cc:5484] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131). New cstate: current_term: 1 leader_uuid: "9f21202c69df478796a5d7dcfe8b07b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:16.457008 19420 raft_consensus.cc:482] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:16.457106 19420 raft_consensus.cc:504] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:16.457500 19420 leader_election.cc:288] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:16.457849 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822ecfcf16fa4c60897cc18d14138289" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662" is_pre_election: true
I20240502 07:58:16.457867 18671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822ecfcf16fa4c60897cc18d14138289" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a" is_pre_election: true
I20240502 07:58:16.457998 18671 raft_consensus.cc:2454] T 822ecfcf16fa4c60897cc18d14138289 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:16.457998 19198 raft_consensus.cc:2454] T 822ecfcf16fa4c60897cc18d14138289 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:16.458281 18868 leader_election.cc:302] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f21202c69df478796a5d7dcfe8b07b5, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:16.458477 19420 raft_consensus.cc:2781] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:16.458521 19420 raft_consensus.cc:482] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:16.458549 19420 raft_consensus.cc:3037] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.459307 19420 raft_consensus.cc:504] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:16.459609 19420 leader_election.cc:288] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Requested vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:16.459945 18671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822ecfcf16fa4c60897cc18d14138289" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a"
I20240502 07:58:16.459951 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "822ecfcf16fa4c60897cc18d14138289" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662"
I20240502 07:58:16.460060 18671 raft_consensus.cc:3037] T 822ecfcf16fa4c60897cc18d14138289 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.460062 19198 raft_consensus.cc:3037] T 822ecfcf16fa4c60897cc18d14138289 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.460965 19198 raft_consensus.cc:2454] T 822ecfcf16fa4c60897cc18d14138289 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
I20240502 07:58:16.460965 18671 raft_consensus.cc:2454] T 822ecfcf16fa4c60897cc18d14138289 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
I20240502 07:58:16.461251 18870 leader_election.cc:302] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, 9f21202c69df478796a5d7dcfe8b07b5; no voters:
I20240502 07:58:16.461447 19420 raft_consensus.cc:2781] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:16.461535 19420 raft_consensus.cc:686] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 LEADER]: Becoming Leader. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Running, Role: LEADER
I20240502 07:58:16.461686 19420 consensus_queue.cc:237] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:16.463348 18527 catalog_manager.cc:5484] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131). New cstate: current_term: 1 leader_uuid: "9f21202c69df478796a5d7dcfe8b07b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:16.516841 19666 raft_consensus.cc:482] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:16.516999 19666 raft_consensus.cc:504] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:16.517478 19666 leader_election.cc:288] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999), 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417)
I20240502 07:58:16.517868 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96cddef766ee4366991da031cbc8e2d5" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" is_pre_election: true
I20240502 07:58:16.517881 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96cddef766ee4366991da031cbc8e2d5" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260" is_pre_election: true
I20240502 07:58:16.518007 19067 raft_consensus.cc:2454] T 96cddef766ee4366991da031cbc8e2d5 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 0.
I20240502 07:58:16.518019 18803 raft_consensus.cc:2454] T 96cddef766ee4366991da031cbc8e2d5 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 0.
I20240502 07:58:16.518301 19264 leader_election.cc:302] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345ab146afdf4d39a1d8583881c06743, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:16.518452 19666 raft_consensus.cc:2781] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:16.518513 19666 raft_consensus.cc:482] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:16.518564 19666 raft_consensus.cc:3037] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.519361 19666 raft_consensus.cc:504] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:16.519703 19666 leader_election.cc:288] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 election: Requested vote from peers 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999), 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417)
I20240502 07:58:16.520087 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96cddef766ee4366991da031cbc8e2d5" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3"
I20240502 07:58:16.520105 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96cddef766ee4366991da031cbc8e2d5" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260"
I20240502 07:58:16.520200 19067 raft_consensus.cc:3037] T 96cddef766ee4366991da031cbc8e2d5 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.520217 18803 raft_consensus.cc:3037] T 96cddef766ee4366991da031cbc8e2d5 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.520941 19067 raft_consensus.cc:2454] T 96cddef766ee4366991da031cbc8e2d5 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 1.
I20240502 07:58:16.521016 18803 raft_consensus.cc:2454] T 96cddef766ee4366991da031cbc8e2d5 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 1.
I20240502 07:58:16.521239 19266 leader_election.cc:302] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 345ab146afdf4d39a1d8583881c06743; no voters:
I20240502 07:58:16.521385 19666 raft_consensus.cc:2781] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:16.521567 19666 raft_consensus.cc:686] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [term 1 LEADER]: Becoming Leader. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Running, Role: LEADER
I20240502 07:58:16.521715 19666 consensus_queue.cc:237] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:16.523447 18527 catalog_manager.cc:5484] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 reported cstate change: term changed from 0 to 1, leader changed from <none> to 345ab146afdf4d39a1d8583881c06743 (127.18.13.134). New cstate: current_term: 1 leader_uuid: "345ab146afdf4d39a1d8583881c06743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:16.673923 19769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:16.674173 19769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:16.677227 19769 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 07:58:16.760382 18671 raft_consensus.cc:1261] T 46247e07dbff4195b8681adb87bccbce P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Refusing update from remote peer 9f21202c69df478796a5d7dcfe8b07b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.760382 19198 raft_consensus.cc:1261] T 46247e07dbff4195b8681adb87bccbce P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Refusing update from remote peer 9f21202c69df478796a5d7dcfe8b07b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.760907 19420 consensus_queue.cc:1035] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.761390 19764 consensus_queue.cc:1035] T 46247e07dbff4195b8681adb87bccbce P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.762626 18803 raft_consensus.cc:1261] T 96cddef766ee4366991da031cbc8e2d5 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Refusing update from remote peer 345ab146afdf4d39a1d8583881c06743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.762698 19067 raft_consensus.cc:1261] T 96cddef766ee4366991da031cbc8e2d5 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Refusing update from remote peer 345ab146afdf4d39a1d8583881c06743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.763142 19767 consensus_queue.cc:1035] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.767778 19767 consensus_queue.cc:1035] T 96cddef766ee4366991da031cbc8e2d5 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.770663 19199 raft_consensus.cc:1261] T 822ecfcf16fa4c60897cc18d14138289 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Refusing update from remote peer 9f21202c69df478796a5d7dcfe8b07b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.770751 18670 raft_consensus.cc:1261] T 822ecfcf16fa4c60897cc18d14138289 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Refusing update from remote peer 9f21202c69df478796a5d7dcfe8b07b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.771049 19764 consensus_queue.cc:1035] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.771335 19420 consensus_queue.cc:1035] T 822ecfcf16fa4c60897cc18d14138289 P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 121.034 ms
time per row: 0.0605168 ms
I20240502 07:58:16.843657 18527 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45150:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:16.849164 19046 tablet_service.cc:1467] Processing CreateTablet for tablet ff4246e864f649fdb53bc67cae2a49a7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=51964072294a48a1a56b25dd7de6a59e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.849457 19046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4246e864f649fdb53bc67cae2a49a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.849490 18782 tablet_service.cc:1467] Processing CreateTablet for tablet ff4246e864f649fdb53bc67cae2a49a7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=51964072294a48a1a56b25dd7de6a59e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.849794 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4246e864f649fdb53bc67cae2a49a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.850463 19178 tablet_service.cc:1467] Processing CreateTablet for tablet 5272d85c6ec1410a9b5e7ab3c0b9bfb2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=51964072294a48a1a56b25dd7de6a59e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.850728 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5272d85c6ec1410a9b5e7ab3c0b9bfb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.851557 19753 tablet_bootstrap.cc:492] T ff4246e864f649fdb53bc67cae2a49a7 P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:16.851845 19310 tablet_service.cc:1467] Processing CreateTablet for tablet ff4246e864f649fdb53bc67cae2a49a7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=51964072294a48a1a56b25dd7de6a59e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.851867 18783 tablet_service.cc:1467] Processing CreateTablet for tablet 431f2dd73e8c42fcb3b5ba171278fbc3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=51964072294a48a1a56b25dd7de6a59e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.852113 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff4246e864f649fdb53bc67cae2a49a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.852116 18783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431f2dd73e8c42fcb3b5ba171278fbc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.852488 19753 tablet_bootstrap.cc:654] T ff4246e864f649fdb53bc67cae2a49a7 P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.853070 19752 tablet_bootstrap.cc:492] T ff4246e864f649fdb53bc67cae2a49a7 P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:16.853893 19752 tablet_bootstrap.cc:654] T ff4246e864f649fdb53bc67cae2a49a7 P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.854205 19748 tablet_bootstrap.cc:492] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:16.854158 18915 tablet_service.cc:1467] Processing CreateTablet for tablet 5272d85c6ec1410a9b5e7ab3c0b9bfb2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=51964072294a48a1a56b25dd7de6a59e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.854395 18915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5272d85c6ec1410a9b5e7ab3c0b9bfb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.855121 19748 tablet_bootstrap.cc:654] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.855381 19753 tablet_bootstrap.cc:492] T ff4246e864f649fdb53bc67cae2a49a7 P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:16.855458 19753 ts_tablet_manager.cc:1367] T ff4246e864f649fdb53bc67cae2a49a7 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 07:58:16.855683 19751 tablet_bootstrap.cc:492] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:16.855916 19753 raft_consensus.cc:348] T ff4246e864f649fdb53bc67cae2a49a7 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:16.856021 19753 raft_consensus.cc:374] T ff4246e864f649fdb53bc67cae2a49a7 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.856055 19753 raft_consensus.cc:729] T ff4246e864f649fdb53bc67cae2a49a7 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.856175 19753 consensus_queue.cc:260] T ff4246e864f649fdb53bc67cae2a49a7 P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:16.856245 19749 tablet_bootstrap.cc:492] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:16.856448 19753 ts_tablet_manager.cc:1398] T ff4246e864f649fdb53bc67cae2a49a7 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.856581 19751 tablet_bootstrap.cc:654] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.856652 19753 tablet_bootstrap.cc:492] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:16.857085 19749 tablet_bootstrap.cc:654] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.857406 19310 tablet_service.cc:1467] Processing CreateTablet for tablet 431f2dd73e8c42fcb3b5ba171278fbc3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=51964072294a48a1a56b25dd7de6a59e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.857497 19753 tablet_bootstrap.cc:654] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.857656 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431f2dd73e8c42fcb3b5ba171278fbc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.857805 19046 tablet_service.cc:1467] Processing CreateTablet for tablet 431f2dd73e8c42fcb3b5ba171278fbc3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=51964072294a48a1a56b25dd7de6a59e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.858033 19046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431f2dd73e8c42fcb3b5ba171278fbc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.859390 18651 tablet_service.cc:1467] Processing CreateTablet for tablet 5272d85c6ec1410a9b5e7ab3c0b9bfb2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=51964072294a48a1a56b25dd7de6a59e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.859668 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5272d85c6ec1410a9b5e7ab3c0b9bfb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.860734 19752 tablet_bootstrap.cc:492] T ff4246e864f649fdb53bc67cae2a49a7 P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:16.860808 19752 ts_tablet_manager.cc:1367] T ff4246e864f649fdb53bc67cae2a49a7 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 07:58:16.861171 19752 raft_consensus.cc:348] T ff4246e864f649fdb53bc67cae2a49a7 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:16.861253 19752 raft_consensus.cc:374] T ff4246e864f649fdb53bc67cae2a49a7 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.861279 19752 raft_consensus.cc:729] T ff4246e864f649fdb53bc67cae2a49a7 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.861409 19748 tablet_bootstrap.cc:492] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:16.861400 19752 consensus_queue.cc:260] T ff4246e864f649fdb53bc67cae2a49a7 P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:16.861507 19748 ts_tablet_manager.cc:1367] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 07:58:16.861640 19752 ts_tablet_manager.cc:1398] T ff4246e864f649fdb53bc67cae2a49a7 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.861778 19752 tablet_bootstrap.cc:492] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:16.861856 19750 tablet_bootstrap.cc:492] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:16.861964 19748 raft_consensus.cc:348] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:16.862056 19748 raft_consensus.cc:374] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.862087 19748 raft_consensus.cc:729] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.862208 19748 consensus_queue.cc:260] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:16.862462 19748 ts_tablet_manager.cc:1398] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.862526 19752 tablet_bootstrap.cc:654] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.862677 19750 tablet_bootstrap.cc:654] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.863662 19749 tablet_bootstrap.cc:492] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:16.863673 19751 tablet_bootstrap.cc:492] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:16.863742 19749 ts_tablet_manager.cc:1367] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240502 07:58:16.863756 19751 ts_tablet_manager.cc:1367] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 07:58:16.864230 19751 raft_consensus.cc:348] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:16.864334 19751 raft_consensus.cc:374] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.864364 19751 raft_consensus.cc:729] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.864470 19751 consensus_queue.cc:260] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:16.864748 19751 ts_tablet_manager.cc:1398] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:16.864904 19751 tablet_bootstrap.cc:492] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:16.865037 19753 tablet_bootstrap.cc:492] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:16.865109 19753 ts_tablet_manager.cc:1367] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 07:58:16.865670 19753 raft_consensus.cc:348] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:16.865748 19751 tablet_bootstrap.cc:654] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.865778 19753 raft_consensus.cc:374] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.865737 19749 raft_consensus.cc:348] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:16.865808 19753 raft_consensus.cc:729] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.865839 19749 raft_consensus.cc:374] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.865866 19749 raft_consensus.cc:729] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.865979 19749 consensus_queue.cc:260] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:16.865969 19753 consensus_queue.cc:260] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:16.866214 19749 ts_tablet_manager.cc:1398] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:16.866408 19753 ts_tablet_manager.cc:1398] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.867532 19750 tablet_bootstrap.cc:492] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:16.867638 19750 ts_tablet_manager.cc:1367] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:58:16.868141 19750 raft_consensus.cc:348] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:16.868243 19750 raft_consensus.cc:374] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.868269 19750 raft_consensus.cc:729] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.868367 19750 consensus_queue.cc:260] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:16.868638 19750 ts_tablet_manager.cc:1398] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.870127 19751 tablet_bootstrap.cc:492] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:16.870204 19751 ts_tablet_manager.cc:1367] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240502 07:58:16.870651 19751 raft_consensus.cc:348] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:16.870764 19751 raft_consensus.cc:374] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.870800 19751 raft_consensus.cc:729] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.870929 19751 consensus_queue.cc:260] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:16.871104 19752 tablet_bootstrap.cc:492] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:16.871160 19751 ts_tablet_manager.cc:1398] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.871188 19752 ts_tablet_manager.cc:1367] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 07:58:16.871637 19752 raft_consensus.cc:348] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:16.871749 19752 raft_consensus.cc:374] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.871822 19752 raft_consensus.cc:729] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.871959 19752 consensus_queue.cc:260] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:16.872243 19752 ts_tablet_manager.cc:1398] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.971480 19808 raft_consensus.cc:482] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:16.971596 19808 raft_consensus.cc:504] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:16.972110 19808 leader_election.cc:288] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417), 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999)
I20240502 07:58:16.972448 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431f2dd73e8c42fcb3b5ba171278fbc3" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" is_pre_election: true
I20240502 07:58:16.972503 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431f2dd73e8c42fcb3b5ba171278fbc3" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260" is_pre_election: true
I20240502 07:58:16.972599 19067 raft_consensus.cc:2454] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 0.
I20240502 07:58:16.972630 18803 raft_consensus.cc:2454] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 0.
I20240502 07:58:16.972879 19266 leader_election.cc:302] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 345ab146afdf4d39a1d8583881c06743; no voters:
I20240502 07:58:16.973039 19808 raft_consensus.cc:2781] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:16.973104 19808 raft_consensus.cc:482] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:16.973140 19808 raft_consensus.cc:3037] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.974376 19808 raft_consensus.cc:504] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:16.974730 19808 leader_election.cc:288] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 election: Requested vote from peers 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417), 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999)
I20240502 07:58:16.975037 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431f2dd73e8c42fcb3b5ba171278fbc3" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3"
I20240502 07:58:16.975145 19067 raft_consensus.cc:3037] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.975438 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431f2dd73e8c42fcb3b5ba171278fbc3" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260"
I20240502 07:58:16.975540 18803 raft_consensus.cc:3037] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.976022 19067 raft_consensus.cc:2454] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 1.
I20240502 07:58:16.976343 18803 raft_consensus.cc:2454] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 1.
I20240502 07:58:16.976625 19264 leader_election.cc:302] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345ab146afdf4d39a1d8583881c06743, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:16.976788 19808 raft_consensus.cc:2781] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:16.976872 19808 raft_consensus.cc:686] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [term 1 LEADER]: Becoming Leader. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Running, Role: LEADER
I20240502 07:58:16.977030 19808 consensus_queue.cc:237] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:16.978731 18520 catalog_manager.cc:5484] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 reported cstate change: term changed from 0 to 1, leader changed from <none> to 345ab146afdf4d39a1d8583881c06743 (127.18.13.134). New cstate: current_term: 1 leader_uuid: "345ab146afdf4d39a1d8583881c06743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:16.996865 19808 raft_consensus.cc:482] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:16.996991 19808 raft_consensus.cc:504] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:16.997412 19808 leader_election.cc:288] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417), 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999)
I20240502 07:58:16.997861 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4246e864f649fdb53bc67cae2a49a7" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" is_pre_election: true
I20240502 07:58:16.997874 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4246e864f649fdb53bc67cae2a49a7" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260" is_pre_election: true
I20240502 07:58:16.998031 19067 raft_consensus.cc:2454] T ff4246e864f649fdb53bc67cae2a49a7 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 0.
I20240502 07:58:16.998036 18803 raft_consensus.cc:2454] T ff4246e864f649fdb53bc67cae2a49a7 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 0.
I20240502 07:58:16.998338 19266 leader_election.cc:302] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 345ab146afdf4d39a1d8583881c06743; no voters:
I20240502 07:58:16.998533 19808 raft_consensus.cc:2781] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:16.998597 19808 raft_consensus.cc:482] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:16.998625 19808 raft_consensus.cc:3037] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.999624 19808 raft_consensus.cc:504] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:16.999991 19808 leader_election.cc:288] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 election: Requested vote from peers 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417), 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999)
I20240502 07:58:17.000311 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4246e864f649fdb53bc67cae2a49a7" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3"
I20240502 07:58:17.000429 19067 raft_consensus.cc:3037] T ff4246e864f649fdb53bc67cae2a49a7 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.000401 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff4246e864f649fdb53bc67cae2a49a7" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260"
I20240502 07:58:17.000558 18803 raft_consensus.cc:3037] T ff4246e864f649fdb53bc67cae2a49a7 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.001408 18803 raft_consensus.cc:2454] T ff4246e864f649fdb53bc67cae2a49a7 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 1.
I20240502 07:58:17.001471 19067 raft_consensus.cc:2454] T ff4246e864f649fdb53bc67cae2a49a7 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 1.
I20240502 07:58:17.001689 19264 leader_election.cc:302] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345ab146afdf4d39a1d8583881c06743, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:17.001900 19808 raft_consensus.cc:2781] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:17.001974 19808 raft_consensus.cc:686] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [term 1 LEADER]: Becoming Leader. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Running, Role: LEADER
I20240502 07:58:17.002121 19808 consensus_queue.cc:237] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.003811 18520 catalog_manager.cc:5484] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 reported cstate change: term changed from 0 to 1, leader changed from <none> to 345ab146afdf4d39a1d8583881c06743 (127.18.13.134). New cstate: current_term: 1 leader_uuid: "345ab146afdf4d39a1d8583881c06743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:17.008558 19804 raft_consensus.cc:482] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:17.008688 19804 raft_consensus.cc:504] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.009052 19804 leader_election.cc:288] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389)
I20240502 07:58:17.009470 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5272d85c6ec1410a9b5e7ab3c0b9bfb2" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662" is_pre_election: true
I20240502 07:58:17.009625 19198 raft_consensus.cc:2454] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 0.
I20240502 07:58:17.009639 18935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5272d85c6ec1410a9b5e7ab3c0b9bfb2" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5" is_pre_election: true
I20240502 07:58:17.009763 18935 raft_consensus.cc:2454] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 0.
I20240502 07:58:17.009964 18604 leader_election.cc:302] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:17.010160 19804 raft_consensus.cc:2781] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:17.010215 19804 raft_consensus.cc:482] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:17.010248 19804 raft_consensus.cc:3037] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.011129 19804 raft_consensus.cc:504] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.011438 19804 leader_election.cc:288] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 election: Requested vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389)
I20240502 07:58:17.011742 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5272d85c6ec1410a9b5e7ab3c0b9bfb2" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662"
I20240502 07:58:17.011879 18935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5272d85c6ec1410a9b5e7ab3c0b9bfb2" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5"
I20240502 07:58:17.011922 19198 raft_consensus.cc:3037] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.011984 18935 raft_consensus.cc:3037] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.012871 18935 raft_consensus.cc:2454] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 1.
I20240502 07:58:17.012871 19198 raft_consensus.cc:2454] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 1.
I20240502 07:58:17.013141 18604 leader_election.cc:302] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:17.013293 19804 raft_consensus.cc:2781] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:17.013383 19804 raft_consensus.cc:686] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [term 1 LEADER]: Becoming Leader. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Running, Role: LEADER
I20240502 07:58:17.013540 19804 consensus_queue.cc:237] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.015272 18520 catalog_manager.cc:5484] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f6bc76c600c413d9f72526563d3234a (127.18.13.129). New cstate: current_term: 1 leader_uuid: "2f6bc76c600c413d9f72526563d3234a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: HEALTHY } } }
W20240502 07:58:17.167975 19823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:17.168221 19823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:17.171193 19823 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 07:58:17.247323 18802 raft_consensus.cc:1261] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Refusing update from remote peer 345ab146afdf4d39a1d8583881c06743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.247326 19067 raft_consensus.cc:1261] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Refusing update from remote peer 345ab146afdf4d39a1d8583881c06743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.247913 19666 consensus_queue.cc:1035] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:17.248561 19808 consensus_queue.cc:1035] T 431f2dd73e8c42fcb3b5ba171278fbc3 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:17.251466 18803 raft_consensus.cc:1261] T ff4246e864f649fdb53bc67cae2a49a7 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Refusing update from remote peer 345ab146afdf4d39a1d8583881c06743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.253446 19066 raft_consensus.cc:1261] T ff4246e864f649fdb53bc67cae2a49a7 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Refusing update from remote peer 345ab146afdf4d39a1d8583881c06743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.254806 19199 raft_consensus.cc:1261] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Refusing update from remote peer 2f6bc76c600c413d9f72526563d3234a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.255120 18935 raft_consensus.cc:1261] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Refusing update from remote peer 2f6bc76c600c413d9f72526563d3234a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.255514 19804 consensus_queue.cc:1035] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:17.255648 19425 consensus_queue.cc:1035] T 5272d85c6ec1410a9b5e7ab3c0b9bfb2 P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:17.256493 19808 consensus_queue.cc:1035] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:17.261705 19666 consensus_queue.cc:1035] T ff4246e864f649fdb53bc67cae2a49a7 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 107.902 ms
time per row: 0.0539512 ms
I20240502 07:58:17.322986 18520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45150:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:17.328568 18915 tablet_service.cc:1467] Processing CreateTablet for tablet 887023183cdc419c9f3d4f6b7bd21fe0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=e5523e05c203477399ddc252c8f52c9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.328940 18915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 887023183cdc419c9f3d4f6b7bd21fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.329380 18651 tablet_service.cc:1467] Processing CreateTablet for tablet 887023183cdc419c9f3d4f6b7bd21fe0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=e5523e05c203477399ddc252c8f52c9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.329370 19046 tablet_service.cc:1467] Processing CreateTablet for tablet 43a36fb737fe418aa662fc75bcf5a3da (DEFAULT_TABLE table=rebalance_test_table_6 [id=e5523e05c203477399ddc252c8f52c9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.329650 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 887023183cdc419c9f3d4f6b7bd21fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.329653 19046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43a36fb737fe418aa662fc75bcf5a3da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.331111 19749 tablet_bootstrap.cc:492] T 887023183cdc419c9f3d4f6b7bd21fe0 P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:17.331274 19750 tablet_bootstrap.cc:492] T 887023183cdc419c9f3d4f6b7bd21fe0 P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:17.331703 19752 tablet_bootstrap.cc:492] T 43a36fb737fe418aa662fc75bcf5a3da P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:17.332046 19749 tablet_bootstrap.cc:654] T 887023183cdc419c9f3d4f6b7bd21fe0 P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.332005 18783 tablet_service.cc:1467] Processing CreateTablet for tablet 43a36fb737fe418aa662fc75bcf5a3da (DEFAULT_TABLE table=rebalance_test_table_6 [id=e5523e05c203477399ddc252c8f52c9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.332149 19750 tablet_bootstrap.cc:654] T 887023183cdc419c9f3d4f6b7bd21fe0 P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.332289 18783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43a36fb737fe418aa662fc75bcf5a3da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.332700 19752 tablet_bootstrap.cc:654] T 43a36fb737fe418aa662fc75bcf5a3da P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.333014 19749 tablet_bootstrap.cc:492] T 887023183cdc419c9f3d4f6b7bd21fe0 P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:17.333086 19749 ts_tablet_manager.cc:1367] T 887023183cdc419c9f3d4f6b7bd21fe0 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:17.333052 19178 tablet_service.cc:1467] Processing CreateTablet for tablet 887023183cdc419c9f3d4f6b7bd21fe0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=e5523e05c203477399ddc252c8f52c9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.333323 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 887023183cdc419c9f3d4f6b7bd21fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.333487 19749 raft_consensus.cc:348] T 887023183cdc419c9f3d4f6b7bd21fe0 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:17.333606 19749 raft_consensus.cc:374] T 887023183cdc419c9f3d4f6b7bd21fe0 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.333637 19749 raft_consensus.cc:729] T 887023183cdc419c9f3d4f6b7bd21fe0 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.333755 19749 consensus_queue.cc:260] T 887023183cdc419c9f3d4f6b7bd21fe0 P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:17.334022 19749 ts_tablet_manager.cc:1398] T 887023183cdc419c9f3d4f6b7bd21fe0 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:17.334430 19753 tablet_bootstrap.cc:492] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:17.335307 19753 tablet_bootstrap.cc:654] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.335402 19748 tablet_bootstrap.cc:492] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:17.335352 18915 tablet_service.cc:1467] Processing CreateTablet for tablet 6a567790cbaf4fc585a0c3276b7ac6d7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=e5523e05c203477399ddc252c8f52c9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.335588 18915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a567790cbaf4fc585a0c3276b7ac6d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.335978 19752 tablet_bootstrap.cc:492] T 43a36fb737fe418aa662fc75bcf5a3da P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:17.336056 19752 ts_tablet_manager.cc:1367] T 43a36fb737fe418aa662fc75bcf5a3da P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:17.336192 19748 tablet_bootstrap.cc:654] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.336535 19752 raft_consensus.cc:348] T 43a36fb737fe418aa662fc75bcf5a3da P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.336658 19752 raft_consensus.cc:374] T 43a36fb737fe418aa662fc75bcf5a3da P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.336695 19752 raft_consensus.cc:729] T 43a36fb737fe418aa662fc75bcf5a3da P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.336827 19752 consensus_queue.cc:260] T 43a36fb737fe418aa662fc75bcf5a3da P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.337551 19310 tablet_service.cc:1467] Processing CreateTablet for tablet 43a36fb737fe418aa662fc75bcf5a3da (DEFAULT_TABLE table=rebalance_test_table_6 [id=e5523e05c203477399ddc252c8f52c9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.337685 19749 tablet_bootstrap.cc:492] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:17.337829 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43a36fb737fe418aa662fc75bcf5a3da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.338109 19752 ts_tablet_manager.cc:1398] T 43a36fb737fe418aa662fc75bcf5a3da P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:17.338907 19749 tablet_bootstrap.cc:654] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.339677 19748 tablet_bootstrap.cc:492] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:17.339772 19748 ts_tablet_manager.cc:1367] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:17.339840 19751 tablet_bootstrap.cc:492] T 43a36fb737fe418aa662fc75bcf5a3da P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:17.340214 19748 raft_consensus.cc:348] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:17.340319 19748 raft_consensus.cc:374] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.340348 19748 raft_consensus.cc:729] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.340466 19748 consensus_queue.cc:260] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:17.340440 18651 tablet_service.cc:1467] Processing CreateTablet for tablet 6a567790cbaf4fc585a0c3276b7ac6d7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=e5523e05c203477399ddc252c8f52c9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.340699 19751 tablet_bootstrap.cc:654] T 43a36fb737fe418aa662fc75bcf5a3da P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.340714 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a567790cbaf4fc585a0c3276b7ac6d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.340749 19748 ts_tablet_manager.cc:1398] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:17.341804 19753 tablet_bootstrap.cc:492] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:17.341899 19753 ts_tablet_manager.cc:1367] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240502 07:58:17.342094 19751 tablet_bootstrap.cc:492] T 43a36fb737fe418aa662fc75bcf5a3da P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:17.342152 19751 ts_tablet_manager.cc:1367] T 43a36fb737fe418aa662fc75bcf5a3da P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:17.342344 19753 raft_consensus.cc:348] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.342449 19753 raft_consensus.cc:374] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.342450 19751 raft_consensus.cc:348] T 43a36fb737fe418aa662fc75bcf5a3da P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.342485 19753 raft_consensus.cc:729] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.342520 19751 raft_consensus.cc:374] T 43a36fb737fe418aa662fc75bcf5a3da P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.342540 19751 raft_consensus.cc:729] T 43a36fb737fe418aa662fc75bcf5a3da P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.342635 19753 consensus_queue.cc:260] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.342650 19751 consensus_queue.cc:260] T 43a36fb737fe418aa662fc75bcf5a3da P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.342900 19753 ts_tablet_manager.cc:1398] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:17.342948 19751 ts_tablet_manager.cc:1398] T 43a36fb737fe418aa662fc75bcf5a3da P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:17.343915 19750 tablet_bootstrap.cc:492] T 887023183cdc419c9f3d4f6b7bd21fe0 P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:17.344018 19750 ts_tablet_manager.cc:1367] T 887023183cdc419c9f3d4f6b7bd21fe0 P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 07:58:17.344616 19750 raft_consensus.cc:348] T 887023183cdc419c9f3d4f6b7bd21fe0 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:17.344725 19750 raft_consensus.cc:374] T 887023183cdc419c9f3d4f6b7bd21fe0 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.344759 19750 raft_consensus.cc:729] T 887023183cdc419c9f3d4f6b7bd21fe0 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.344877 19750 consensus_queue.cc:260] T 887023183cdc419c9f3d4f6b7bd21fe0 P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:17.345153 19750 ts_tablet_manager.cc:1398] T 887023183cdc419c9f3d4f6b7bd21fe0 P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:17.345371 19750 tablet_bootstrap.cc:492] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:17.345580 19178 tablet_service.cc:1467] Processing CreateTablet for tablet 6a567790cbaf4fc585a0c3276b7ac6d7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=e5523e05c203477399ddc252c8f52c9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.345816 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a567790cbaf4fc585a0c3276b7ac6d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.347421 19750 tablet_bootstrap.cc:654] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.348763 19750 tablet_bootstrap.cc:492] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:17.348771 19749 tablet_bootstrap.cc:492] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:17.348846 19750 ts_tablet_manager.cc:1367] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 07:58:17.348902 19748 tablet_bootstrap.cc:492] T 6a567790cbaf4fc585a0c3276b7ac6d7 P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:17.349292 19750 raft_consensus.cc:348] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.349391 19750 raft_consensus.cc:374] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.349424 19750 raft_consensus.cc:729] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.349510 19749 ts_tablet_manager.cc:1367] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 07:58:17.349529 19750 consensus_queue.cc:260] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.349764 19748 tablet_bootstrap.cc:654] T 6a567790cbaf4fc585a0c3276b7ac6d7 P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.349779 19750 ts_tablet_manager.cc:1398] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:17.349985 19749 raft_consensus.cc:348] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.350229 19749 raft_consensus.cc:374] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.350294 19749 raft_consensus.cc:729] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.350459 19749 consensus_queue.cc:260] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.350857 19749 ts_tablet_manager.cc:1398] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:17.350999 19748 tablet_bootstrap.cc:492] T 6a567790cbaf4fc585a0c3276b7ac6d7 P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:17.351079 19748 ts_tablet_manager.cc:1367] T 6a567790cbaf4fc585a0c3276b7ac6d7 P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 07:58:17.351423 19748 raft_consensus.cc:348] T 6a567790cbaf4fc585a0c3276b7ac6d7 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.351536 19748 raft_consensus.cc:374] T 6a567790cbaf4fc585a0c3276b7ac6d7 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.351576 19748 raft_consensus.cc:729] T 6a567790cbaf4fc585a0c3276b7ac6d7 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.351702 19748 consensus_queue.cc:260] T 6a567790cbaf4fc585a0c3276b7ac6d7 P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.351965 19748 ts_tablet_manager.cc:1398] T 6a567790cbaf4fc585a0c3276b7ac6d7 P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:17.358366 19865 raft_consensus.cc:482] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:17.358444 19865 raft_consensus.cc:504] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.358819 19865 leader_election.cc:288] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389)
I20240502 07:58:17.359102 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a567790cbaf4fc585a0c3276b7ac6d7" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662" is_pre_election: true
I20240502 07:58:17.359172 18935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a567790cbaf4fc585a0c3276b7ac6d7" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5" is_pre_election: true
I20240502 07:58:17.359225 19199 raft_consensus.cc:2454] T 6a567790cbaf4fc585a0c3276b7ac6d7 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 0.
I20240502 07:58:17.359297 18935 raft_consensus.cc:2454] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 0.
I20240502 07:58:17.359551 18605 leader_election.cc:302] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, 9f21202c69df478796a5d7dcfe8b07b5; no voters:
I20240502 07:58:17.359702 19865 raft_consensus.cc:2781] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:17.359762 19865 raft_consensus.cc:482] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:17.359827 19865 raft_consensus.cc:3037] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.360581 19865 raft_consensus.cc:504] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.360857 19865 leader_election.cc:288] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 election: Requested vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389)
I20240502 07:58:17.361128 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a567790cbaf4fc585a0c3276b7ac6d7" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662"
I20240502 07:58:17.361178 18935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a567790cbaf4fc585a0c3276b7ac6d7" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5"
I20240502 07:58:17.361225 19199 raft_consensus.cc:3037] T 6a567790cbaf4fc585a0c3276b7ac6d7 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.361282 18935 raft_consensus.cc:3037] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.361907 19199 raft_consensus.cc:2454] T 6a567790cbaf4fc585a0c3276b7ac6d7 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 1.
I20240502 07:58:17.361991 18935 raft_consensus.cc:2454] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 1.
I20240502 07:58:17.362174 18604 leader_election.cc:302] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:17.362378 19865 raft_consensus.cc:2781] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:17.362445 19865 raft_consensus.cc:686] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [term 1 LEADER]: Becoming Leader. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Running, Role: LEADER
I20240502 07:58:17.362588 19865 consensus_queue.cc:237] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.364141 18529 catalog_manager.cc:5484] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f6bc76c600c413d9f72526563d3234a (127.18.13.129). New cstate: current_term: 1 leader_uuid: "2f6bc76c600c413d9f72526563d3234a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:17.396234 19723 raft_consensus.cc:482] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:17.396361 19723 raft_consensus.cc:504] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:17.396900 19723 leader_election.cc:288] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:17.397256 18934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887023183cdc419c9f3d4f6b7bd21fe0" candidate_uuid: "d19f943e214647a3850f51a324017662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5" is_pre_election: true
I20240502 07:58:17.397310 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887023183cdc419c9f3d4f6b7bd21fe0" candidate_uuid: "d19f943e214647a3850f51a324017662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a" is_pre_election: true
I20240502 07:58:17.397385 18934 raft_consensus.cc:2454] T 887023183cdc419c9f3d4f6b7bd21fe0 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d19f943e214647a3850f51a324017662 in term 0.
I20240502 07:58:17.397455 18670 raft_consensus.cc:2454] T 887023183cdc419c9f3d4f6b7bd21fe0 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d19f943e214647a3850f51a324017662 in term 0.
I20240502 07:58:17.397661 19133 leader_election.cc:302] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f21202c69df478796a5d7dcfe8b07b5, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:17.397966 19723 raft_consensus.cc:2781] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:17.398031 19723 raft_consensus.cc:482] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:17.398070 19723 raft_consensus.cc:3037] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.399055 19723 raft_consensus.cc:504] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:17.399363 19723 leader_election.cc:288] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [CANDIDATE]: Term 1 election: Requested vote from peers 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:17.399663 18934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887023183cdc419c9f3d4f6b7bd21fe0" candidate_uuid: "d19f943e214647a3850f51a324017662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5"
I20240502 07:58:17.399686 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887023183cdc419c9f3d4f6b7bd21fe0" candidate_uuid: "d19f943e214647a3850f51a324017662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a"
I20240502 07:58:17.399806 18670 raft_consensus.cc:3037] T 887023183cdc419c9f3d4f6b7bd21fe0 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.399804 18934 raft_consensus.cc:3037] T 887023183cdc419c9f3d4f6b7bd21fe0 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.400746 18670 raft_consensus.cc:2454] T 887023183cdc419c9f3d4f6b7bd21fe0 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d19f943e214647a3850f51a324017662 in term 1.
I20240502 07:58:17.400746 18934 raft_consensus.cc:2454] T 887023183cdc419c9f3d4f6b7bd21fe0 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d19f943e214647a3850f51a324017662 in term 1.
I20240502 07:58:17.401026 19133 leader_election.cc:302] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f21202c69df478796a5d7dcfe8b07b5, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:17.401175 19723 raft_consensus.cc:2781] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:17.401342 19723 raft_consensus.cc:686] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [term 1 LEADER]: Becoming Leader. State: Replica: d19f943e214647a3850f51a324017662, State: Running, Role: LEADER
I20240502 07:58:17.401700 19723 consensus_queue.cc:237] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:17.403470 18529 catalog_manager.cc:5484] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 reported cstate change: term changed from 0 to 1, leader changed from <none> to d19f943e214647a3850f51a324017662 (127.18.13.133). New cstate: current_term: 1 leader_uuid: "d19f943e214647a3850f51a324017662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:17.534549 19856 raft_consensus.cc:482] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:17.534682 19856 raft_consensus.cc:504] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.535032 19856 leader_election.cc:288] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417), 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647)
I20240502 07:58:17.535362 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a36fb737fe418aa662fc75bcf5a3da" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" is_pre_election: true
I20240502 07:58:17.535442 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a36fb737fe418aa662fc75bcf5a3da" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743" is_pre_election: true
I20240502 07:58:17.535497 19067 raft_consensus.cc:2454] T 43a36fb737fe418aa662fc75bcf5a3da P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 0.
I20240502 07:58:17.535599 19331 raft_consensus.cc:2454] T 43a36fb737fe418aa662fc75bcf5a3da P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 0.
I20240502 07:58:17.535849 18738 leader_election.cc:302] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:17.536036 19856 raft_consensus.cc:2781] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:17.536098 19856 raft_consensus.cc:482] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:17.536123 19856 raft_consensus.cc:3037] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.537010 19856 raft_consensus.cc:504] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.537320 19856 leader_election.cc:288] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 election: Requested vote from peers 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417), 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647)
I20240502 07:58:17.537637 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a36fb737fe418aa662fc75bcf5a3da" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3"
I20240502 07:58:17.537744 19067 raft_consensus.cc:3037] T 43a36fb737fe418aa662fc75bcf5a3da P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.537703 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a36fb737fe418aa662fc75bcf5a3da" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743"
I20240502 07:58:17.537837 19331 raft_consensus.cc:3037] T 43a36fb737fe418aa662fc75bcf5a3da P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.538514 19067 raft_consensus.cc:2454] T 43a36fb737fe418aa662fc75bcf5a3da P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 1.
I20240502 07:58:17.538666 19331 raft_consensus.cc:2454] T 43a36fb737fe418aa662fc75bcf5a3da P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 1.
I20240502 07:58:17.538787 18738 leader_election.cc:302] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:17.538961 19856 raft_consensus.cc:2781] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:17.539043 19856 raft_consensus.cc:686] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 LEADER]: Becoming Leader. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Running, Role: LEADER
I20240502 07:58:17.539177 19856 consensus_queue.cc:237] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.541079 18520 catalog_manager.cc:5484] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "4135e6d90aec4830b08bcb7e8dc50260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:17.671424 19879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:17.671675 19879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:17.675136 19879 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 07:58:17.771517 18670 raft_consensus.cc:1261] T 887023183cdc419c9f3d4f6b7bd21fe0 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Refusing update from remote peer d19f943e214647a3850f51a324017662: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.772143 19723 consensus_queue.cc:1035] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:17.775151 18934 raft_consensus.cc:1261] T 887023183cdc419c9f3d4f6b7bd21fe0 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Refusing update from remote peer d19f943e214647a3850f51a324017662: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.776350 19067 raft_consensus.cc:1261] T 43a36fb737fe418aa662fc75bcf5a3da P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Refusing update from remote peer 4135e6d90aec4830b08bcb7e8dc50260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.776921 19856 consensus_queue.cc:1035] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:17.778183 18934 raft_consensus.cc:1261] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Refusing update from remote peer 2f6bc76c600c413d9f72526563d3234a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.778717 19331 raft_consensus.cc:1261] T 43a36fb737fe418aa662fc75bcf5a3da P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Refusing update from remote peer 4135e6d90aec4830b08bcb7e8dc50260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.779071 19865 consensus_queue.cc:1035] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:17.779101 19856 consensus_queue.cc:1035] T 43a36fb737fe418aa662fc75bcf5a3da P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:17.781453 19865 consensus_queue.cc:1035] T 6a567790cbaf4fc585a0c3276b7ac6d7 P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:17.781826 19723 consensus_queue.cc:1035] T 887023183cdc419c9f3d4f6b7bd21fe0 P d19f943e214647a3850f51a324017662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 138.037 ms
time per row: 0.0690183 ms
I20240502 07:58:17.856874 18520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45150:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:17.862761 19046 tablet_service.cc:1467] Processing CreateTablet for tablet 518afbe60085421aa353b7a7ac13d4d9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0ccfedf158a441ac920f000344df9c17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.863032 18782 tablet_service.cc:1467] Processing CreateTablet for tablet 275d344bb7e044db9f38f9d1215a0c2a (DEFAULT_TABLE table=rebalance_test_table_7 [id=0ccfedf158a441ac920f000344df9c17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.863133 19046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 518afbe60085421aa353b7a7ac13d4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.863152 19047 tablet_service.cc:1467] Processing CreateTablet for tablet 275d344bb7e044db9f38f9d1215a0c2a (DEFAULT_TABLE table=rebalance_test_table_7 [id=0ccfedf158a441ac920f000344df9c17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.863322 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275d344bb7e044db9f38f9d1215a0c2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.863344 19047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275d344bb7e044db9f38f9d1215a0c2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.863602 18915 tablet_service.cc:1467] Processing CreateTablet for tablet e128121cf15a4b5aa3f07265ed60371f (DEFAULT_TABLE table=rebalance_test_table_7 [id=0ccfedf158a441ac920f000344df9c17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.863755 18783 tablet_service.cc:1467] Processing CreateTablet for tablet 518afbe60085421aa353b7a7ac13d4d9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0ccfedf158a441ac920f000344df9c17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.863894 18915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e128121cf15a4b5aa3f07265ed60371f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.863988 18783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 518afbe60085421aa353b7a7ac13d4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.866088 18651 tablet_service.cc:1467] Processing CreateTablet for tablet e128121cf15a4b5aa3f07265ed60371f (DEFAULT_TABLE table=rebalance_test_table_7 [id=0ccfedf158a441ac920f000344df9c17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.866125 19310 tablet_service.cc:1467] Processing CreateTablet for tablet 518afbe60085421aa353b7a7ac13d4d9 (DEFAULT_TABLE table=rebalance_test_table_7 [id=0ccfedf158a441ac920f000344df9c17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.866295 19925 tablet_bootstrap.cc:492] T 275d344bb7e044db9f38f9d1215a0c2a P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:17.866358 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e128121cf15a4b5aa3f07265ed60371f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.866369 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 518afbe60085421aa353b7a7ac13d4d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.866973 19925 tablet_bootstrap.cc:654] T 275d344bb7e044db9f38f9d1215a0c2a P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.868005 19925 tablet_bootstrap.cc:492] T 275d344bb7e044db9f38f9d1215a0c2a P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:17.868083 19925 ts_tablet_manager.cc:1367] T 275d344bb7e044db9f38f9d1215a0c2a P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:17.868566 19925 raft_consensus.cc:348] T 275d344bb7e044db9f38f9d1215a0c2a P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.868665 19925 raft_consensus.cc:374] T 275d344bb7e044db9f38f9d1215a0c2a P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.868669 19929 tablet_bootstrap.cc:492] T e128121cf15a4b5aa3f07265ed60371f P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:17.868693 19925 raft_consensus.cc:729] T 275d344bb7e044db9f38f9d1215a0c2a P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.868809 19925 consensus_queue.cc:260] T 275d344bb7e044db9f38f9d1215a0c2a P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.869113 19925 ts_tablet_manager.cc:1398] T 275d344bb7e044db9f38f9d1215a0c2a P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:17.869253 19925 tablet_bootstrap.cc:492] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:17.869438 19926 tablet_bootstrap.cc:492] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:17.870151 19924 tablet_bootstrap.cc:492] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:17.870183 19310 tablet_service.cc:1467] Processing CreateTablet for tablet 275d344bb7e044db9f38f9d1215a0c2a (DEFAULT_TABLE table=rebalance_test_table_7 [id=0ccfedf158a441ac920f000344df9c17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.870425 19925 tablet_bootstrap.cc:654] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.870455 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275d344bb7e044db9f38f9d1215a0c2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.871100 19924 tablet_bootstrap.cc:654] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.871661 19926 tablet_bootstrap.cc:654] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.872246 19178 tablet_service.cc:1467] Processing CreateTablet for tablet e128121cf15a4b5aa3f07265ed60371f (DEFAULT_TABLE table=rebalance_test_table_7 [id=0ccfedf158a441ac920f000344df9c17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:17.872514 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e128121cf15a4b5aa3f07265ed60371f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:17.873147 19926 tablet_bootstrap.cc:492] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:17.873224 19926 ts_tablet_manager.cc:1367] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:17.873689 19926 raft_consensus.cc:348] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.873800 19926 raft_consensus.cc:374] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.873831 19926 raft_consensus.cc:729] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.874015 19926 consensus_queue.cc:260] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.874380 19926 ts_tablet_manager.cc:1398] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:17.874995 19929 tablet_bootstrap.cc:654] T e128121cf15a4b5aa3f07265ed60371f P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.875113 19933 tablet_bootstrap.cc:492] T e128121cf15a4b5aa3f07265ed60371f P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:17.875718 19928 tablet_bootstrap.cc:492] T 518afbe60085421aa353b7a7ac13d4d9 P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:17.876533 19928 tablet_bootstrap.cc:654] T 518afbe60085421aa353b7a7ac13d4d9 P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.876714 19933 tablet_bootstrap.cc:654] T e128121cf15a4b5aa3f07265ed60371f P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.876868 19929 tablet_bootstrap.cc:492] T e128121cf15a4b5aa3f07265ed60371f P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:17.877063 19929 ts_tablet_manager.cc:1367] T e128121cf15a4b5aa3f07265ed60371f P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 07:58:17.877868 19924 tablet_bootstrap.cc:492] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:17.877897 19928 tablet_bootstrap.cc:492] T 518afbe60085421aa353b7a7ac13d4d9 P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:17.877971 19924 ts_tablet_manager.cc:1367] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240502 07:58:17.877981 19928 ts_tablet_manager.cc:1367] T 518afbe60085421aa353b7a7ac13d4d9 P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:17.878314 19928 raft_consensus.cc:348] T 518afbe60085421aa353b7a7ac13d4d9 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.878415 19928 raft_consensus.cc:374] T 518afbe60085421aa353b7a7ac13d4d9 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.878451 19928 raft_consensus.cc:729] T 518afbe60085421aa353b7a7ac13d4d9 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.878410 19929 raft_consensus.cc:348] T e128121cf15a4b5aa3f07265ed60371f P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.878513 19929 raft_consensus.cc:374] T e128121cf15a4b5aa3f07265ed60371f P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.878564 19929 raft_consensus.cc:729] T e128121cf15a4b5aa3f07265ed60371f P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.878563 19924 raft_consensus.cc:348] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.878608 19928 consensus_queue.cc:260] T 518afbe60085421aa353b7a7ac13d4d9 P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.878669 19924 raft_consensus.cc:374] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.878700 19924 raft_consensus.cc:729] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.878705 19929 consensus_queue.cc:260] T e128121cf15a4b5aa3f07265ed60371f P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.878823 19924 consensus_queue.cc:260] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.878991 19929 ts_tablet_manager.cc:1398] T e128121cf15a4b5aa3f07265ed60371f P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 07:58:17.879071 19924 ts_tablet_manager.cc:1398] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:17.879220 19924 tablet_bootstrap.cc:492] T 518afbe60085421aa353b7a7ac13d4d9 P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:17.879482 19925 tablet_bootstrap.cc:492] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:17.879563 19925 ts_tablet_manager.cc:1367] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240502 07:58:17.879940 19928 ts_tablet_manager.cc:1398] T 518afbe60085421aa353b7a7ac13d4d9 P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:17.880100 19928 tablet_bootstrap.cc:492] T 275d344bb7e044db9f38f9d1215a0c2a P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:17.880146 19924 tablet_bootstrap.cc:654] T 518afbe60085421aa353b7a7ac13d4d9 P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.880621 19925 raft_consensus.cc:348] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.880805 19925 raft_consensus.cc:374] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.880904 19925 raft_consensus.cc:729] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.881072 19925 consensus_queue.cc:260] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.881282 19933 tablet_bootstrap.cc:492] T e128121cf15a4b5aa3f07265ed60371f P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:17.881394 19933 ts_tablet_manager.cc:1367] T e128121cf15a4b5aa3f07265ed60371f P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 07:58:17.881628 19925 ts_tablet_manager.cc:1398] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:17.883229 19933 raft_consensus.cc:348] T e128121cf15a4b5aa3f07265ed60371f P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.883476 19933 raft_consensus.cc:374] T e128121cf15a4b5aa3f07265ed60371f P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.883581 19933 raft_consensus.cc:729] T e128121cf15a4b5aa3f07265ed60371f P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.880897 19928 tablet_bootstrap.cc:654] T 275d344bb7e044db9f38f9d1215a0c2a P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:17.884569 19933 consensus_queue.cc:260] T e128121cf15a4b5aa3f07265ed60371f P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.884676 19924 tablet_bootstrap.cc:492] T 518afbe60085421aa353b7a7ac13d4d9 P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:17.884766 19924 ts_tablet_manager.cc:1367] T 518afbe60085421aa353b7a7ac13d4d9 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:58:17.884925 19928 tablet_bootstrap.cc:492] T 275d344bb7e044db9f38f9d1215a0c2a P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:17.885015 19928 ts_tablet_manager.cc:1367] T 275d344bb7e044db9f38f9d1215a0c2a P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 07:58:17.885141 19924 raft_consensus.cc:348] T 518afbe60085421aa353b7a7ac13d4d9 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.885231 19924 raft_consensus.cc:374] T 518afbe60085421aa353b7a7ac13d4d9 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.885272 19924 raft_consensus.cc:729] T 518afbe60085421aa353b7a7ac13d4d9 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.885408 19924 consensus_queue.cc:260] T 518afbe60085421aa353b7a7ac13d4d9 P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.885452 19928 raft_consensus.cc:348] T 275d344bb7e044db9f38f9d1215a0c2a P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.885581 19928 raft_consensus.cc:374] T 275d344bb7e044db9f38f9d1215a0c2a P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:17.885635 19928 raft_consensus.cc:729] T 275d344bb7e044db9f38f9d1215a0c2a P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:17.885684 19924 ts_tablet_manager.cc:1398] T 518afbe60085421aa353b7a7ac13d4d9 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:17.885779 19928 consensus_queue.cc:260] T 275d344bb7e044db9f38f9d1215a0c2a P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.886291 19928 ts_tablet_manager.cc:1398] T 275d344bb7e044db9f38f9d1215a0c2a P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:17.886641 19933 ts_tablet_manager.cc:1398] T e128121cf15a4b5aa3f07265ed60371f P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 07:58:17.892282 19806 raft_consensus.cc:482] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:17.892378 19806 raft_consensus.cc:504] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.892697 19806 leader_election.cc:288] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:17.893071 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e128121cf15a4b5aa3f07265ed60371f" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662" is_pre_election: true
I20240502 07:58:17.893074 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e128121cf15a4b5aa3f07265ed60371f" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a" is_pre_election: true
I20240502 07:58:17.893203 19199 raft_consensus.cc:2454] T e128121cf15a4b5aa3f07265ed60371f P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:17.893203 18670 raft_consensus.cc:2454] T e128121cf15a4b5aa3f07265ed60371f P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:17.893520 18870 leader_election.cc:302] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, 9f21202c69df478796a5d7dcfe8b07b5; no voters:
I20240502 07:58:17.893780 19806 raft_consensus.cc:2781] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:17.893843 19806 raft_consensus.cc:482] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:17.893886 19806 raft_consensus.cc:3037] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.894796 19806 raft_consensus.cc:504] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.895114 19806 leader_election.cc:288] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Requested vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:17.895466 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e128121cf15a4b5aa3f07265ed60371f" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a"
I20240502 07:58:17.895596 18670 raft_consensus.cc:3037] T e128121cf15a4b5aa3f07265ed60371f P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.896498 18670 raft_consensus.cc:2454] T e128121cf15a4b5aa3f07265ed60371f P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
I20240502 07:58:17.896720 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e128121cf15a4b5aa3f07265ed60371f" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662"
I20240502 07:58:17.896809 19199 raft_consensus.cc:3037] T e128121cf15a4b5aa3f07265ed60371f P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.896788 18870 leader_election.cc:302] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, 9f21202c69df478796a5d7dcfe8b07b5; no voters:
I20240502 07:58:17.896972 19806 raft_consensus.cc:2781] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:17.897085 19806 raft_consensus.cc:686] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 LEADER]: Becoming Leader. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Running, Role: LEADER
I20240502 07:58:17.897212 19806 consensus_queue.cc:237] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:17.897758 19199 raft_consensus.cc:2454] T e128121cf15a4b5aa3f07265ed60371f P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
I20240502 07:58:17.898960 18523 catalog_manager.cc:5484] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131). New cstate: current_term: 1 leader_uuid: "9f21202c69df478796a5d7dcfe8b07b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:17.929997 19824 raft_consensus.cc:482] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:17.930150 19824 raft_consensus.cc:504] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.930605 19824 leader_election.cc:288] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999), 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647)
I20240502 07:58:17.931099 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518afbe60085421aa353b7a7ac13d4d9" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743" is_pre_election: true
I20240502 07:58:17.931099 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518afbe60085421aa353b7a7ac13d4d9" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260" is_pre_election: true
I20240502 07:58:17.931252 19331 raft_consensus.cc:2454] T 518afbe60085421aa353b7a7ac13d4d9 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 0.
I20240502 07:58:17.931252 18803 raft_consensus.cc:2454] T 518afbe60085421aa353b7a7ac13d4d9 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 0.
I20240502 07:58:17.931550 19000 leader_election.cc:302] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 345ab146afdf4d39a1d8583881c06743; no voters:
I20240502 07:58:17.931717 19824 raft_consensus.cc:2781] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:17.931779 19824 raft_consensus.cc:482] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:17.931850 19824 raft_consensus.cc:3037] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.932931 19824 raft_consensus.cc:504] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.933295 19824 leader_election.cc:288] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 election: Requested vote from peers 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999), 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647)
I20240502 07:58:17.933629 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518afbe60085421aa353b7a7ac13d4d9" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260"
I20240502 07:58:17.933743 18803 raft_consensus.cc:3037] T 518afbe60085421aa353b7a7ac13d4d9 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.933689 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "518afbe60085421aa353b7a7ac13d4d9" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743"
I20240502 07:58:17.933830 19331 raft_consensus.cc:3037] T 518afbe60085421aa353b7a7ac13d4d9 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.934501 18803 raft_consensus.cc:2454] T 518afbe60085421aa353b7a7ac13d4d9 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 1.
I20240502 07:58:17.934621 19331 raft_consensus.cc:2454] T 518afbe60085421aa353b7a7ac13d4d9 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 1.
I20240502 07:58:17.934788 19000 leader_election.cc:302] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:17.934948 19824 raft_consensus.cc:2781] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:17.935130 19824 raft_consensus.cc:686] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 LEADER]: Becoming Leader. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Running, Role: LEADER
I20240502 07:58:17.935271 19824 consensus_queue.cc:237] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.936980 18523 catalog_manager.cc:5484] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132). New cstate: current_term: 1 leader_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:17.965939 19856 raft_consensus.cc:482] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:17.966060 19856 raft_consensus.cc:504] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.966473 19856 leader_election.cc:288] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417), 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647)
I20240502 07:58:17.966918 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275d344bb7e044db9f38f9d1215a0c2a" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743" is_pre_election: true
I20240502 07:58:17.966917 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275d344bb7e044db9f38f9d1215a0c2a" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" is_pre_election: true
I20240502 07:58:17.967065 19331 raft_consensus.cc:2454] T 275d344bb7e044db9f38f9d1215a0c2a P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 0.
I20240502 07:58:17.967075 19067 raft_consensus.cc:2454] T 275d344bb7e044db9f38f9d1215a0c2a P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 0.
I20240502 07:58:17.967394 18736 leader_election.cc:302] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345ab146afdf4d39a1d8583881c06743, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:17.967579 19856 raft_consensus.cc:2781] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:17.967633 19856 raft_consensus.cc:482] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:17.967669 19856 raft_consensus.cc:3037] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.968683 19856 raft_consensus.cc:504] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.968979 19856 leader_election.cc:288] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 election: Requested vote from peers 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417), 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647)
I20240502 07:58:17.969291 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275d344bb7e044db9f38f9d1215a0c2a" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743"
I20240502 07:58:17.969291 19067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275d344bb7e044db9f38f9d1215a0c2a" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3"
I20240502 07:58:17.969404 19067 raft_consensus.cc:3037] T 275d344bb7e044db9f38f9d1215a0c2a P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.969404 19331 raft_consensus.cc:3037] T 275d344bb7e044db9f38f9d1215a0c2a P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:17.970320 19067 raft_consensus.cc:2454] T 275d344bb7e044db9f38f9d1215a0c2a P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 1.
I20240502 07:58:17.970320 19331 raft_consensus.cc:2454] T 275d344bb7e044db9f38f9d1215a0c2a P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 1.
I20240502 07:58:17.970604 18736 leader_election.cc:302] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345ab146afdf4d39a1d8583881c06743, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:17.970748 19856 raft_consensus.cc:2781] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:17.970853 19856 raft_consensus.cc:686] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 LEADER]: Becoming Leader. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Running, Role: LEADER
I20240502 07:58:17.971015 19856 consensus_queue.cc:237] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } }
I20240502 07:58:17.972702 18523 catalog_manager.cc:5484] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "4135e6d90aec4830b08bcb7e8dc50260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:18.108305 19943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:18.108599 19943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:18.111745 19943 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 07:58:18.189348 19331 raft_consensus.cc:1261] T 275d344bb7e044db9f38f9d1215a0c2a P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Refusing update from remote peer 4135e6d90aec4830b08bcb7e8dc50260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.189752 19856 consensus_queue.cc:1035] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:18.191454 19330 raft_consensus.cc:1261] T 518afbe60085421aa353b7a7ac13d4d9 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Refusing update from remote peer 341d0eb74c3544faaa2633d7aa3f77b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.191955 19940 consensus_queue.cc:1035] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:18.194444 18670 raft_consensus.cc:1261] T e128121cf15a4b5aa3f07265ed60371f P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Refusing update from remote peer 9f21202c69df478796a5d7dcfe8b07b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.195029 19199 raft_consensus.cc:1261] T e128121cf15a4b5aa3f07265ed60371f P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Refusing update from remote peer 9f21202c69df478796a5d7dcfe8b07b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.194950 19806 consensus_queue.cc:1035] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:18.195600 18803 raft_consensus.cc:1261] T 518afbe60085421aa353b7a7ac13d4d9 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Refusing update from remote peer 341d0eb74c3544faaa2633d7aa3f77b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.196153 19067 raft_consensus.cc:1261] T 275d344bb7e044db9f38f9d1215a0c2a P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Refusing update from remote peer 4135e6d90aec4830b08bcb7e8dc50260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.196856 19856 consensus_queue.cc:1035] T 275d344bb7e044db9f38f9d1215a0c2a P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:18.198182 19940 consensus_queue.cc:1035] T 518afbe60085421aa353b7a7ac13d4d9 P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:18.200176 19806 consensus_queue.cc:1035] T e128121cf15a4b5aa3f07265ed60371f P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.843 ms
time per row: 0.0569215 ms
I20240502 07:58:18.272246 18520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45150:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:18.277645 18651 tablet_service.cc:1467] Processing CreateTablet for tablet 84aed0b3a5e74500b8646d0dbe9b26a1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d38382222f14478fb52d57185405af94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.277984 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84aed0b3a5e74500b8646d0dbe9b26a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.278147 18783 tablet_service.cc:1467] Processing CreateTablet for tablet 08f66fc67af845b08eaf00717e75d667 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d38382222f14478fb52d57185405af94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.278404 18783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f66fc67af845b08eaf00717e75d667. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.279657 19929 tablet_bootstrap.cc:492] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:18.280098 19046 tablet_service.cc:1467] Processing CreateTablet for tablet 08f66fc67af845b08eaf00717e75d667 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d38382222f14478fb52d57185405af94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.280297 19310 tablet_service.cc:1467] Processing CreateTablet for tablet 08f66fc67af845b08eaf00717e75d667 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d38382222f14478fb52d57185405af94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.280350 19046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f66fc67af845b08eaf00717e75d667. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.280508 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08f66fc67af845b08eaf00717e75d667. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.280545 19929 tablet_bootstrap.cc:654] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.281709 19929 tablet_bootstrap.cc:492] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:18.281782 19929 ts_tablet_manager.cc:1367] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:18.282161 19928 tablet_bootstrap.cc:492] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:18.282233 19924 tablet_bootstrap.cc:492] T 08f66fc67af845b08eaf00717e75d667 P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:18.282251 19929 raft_consensus.cc:348] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.282358 19929 raft_consensus.cc:374] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.282389 19929 raft_consensus.cc:729] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.282523 19929 consensus_queue.cc:260] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.282559 18650 tablet_service.cc:1467] Processing CreateTablet for tablet c7376645e76b4c5abdcd721bea1dae5d (DEFAULT_TABLE table=rebalance_test_table_8 [id=d38382222f14478fb52d57185405af94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.282760 18650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7376645e76b4c5abdcd721bea1dae5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.282819 19929 ts_tablet_manager.cc:1398] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:18.283026 19924 tablet_bootstrap.cc:654] T 08f66fc67af845b08eaf00717e75d667 P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.283296 19928 tablet_bootstrap.cc:654] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.284533 19925 tablet_bootstrap.cc:492] T 08f66fc67af845b08eaf00717e75d667 P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:18.284489 19179 tablet_service.cc:1467] Processing CreateTablet for tablet c7376645e76b4c5abdcd721bea1dae5d (DEFAULT_TABLE table=rebalance_test_table_8 [id=d38382222f14478fb52d57185405af94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.284562 19924 tablet_bootstrap.cc:492] T 08f66fc67af845b08eaf00717e75d667 P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:18.284621 19924 ts_tablet_manager.cc:1367] T 08f66fc67af845b08eaf00717e75d667 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:18.284734 19179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7376645e76b4c5abdcd721bea1dae5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.284960 19924 raft_consensus.cc:348] T 08f66fc67af845b08eaf00717e75d667 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:18.285049 19924 raft_consensus.cc:374] T 08f66fc67af845b08eaf00717e75d667 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.285071 19924 raft_consensus.cc:729] T 08f66fc67af845b08eaf00717e75d667 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.285185 19924 consensus_queue.cc:260] T 08f66fc67af845b08eaf00717e75d667 P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:18.285367 19925 tablet_bootstrap.cc:654] T 08f66fc67af845b08eaf00717e75d667 P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.285432 19924 ts_tablet_manager.cc:1398] T 08f66fc67af845b08eaf00717e75d667 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:18.286062 19928 tablet_bootstrap.cc:492] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:18.286135 19928 ts_tablet_manager.cc:1367] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 07:58:18.286379 19925 tablet_bootstrap.cc:492] T 08f66fc67af845b08eaf00717e75d667 P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:18.286442 19929 tablet_bootstrap.cc:492] T c7376645e76b4c5abdcd721bea1dae5d P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:18.286456 19925 ts_tablet_manager.cc:1367] T 08f66fc67af845b08eaf00717e75d667 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:18.286617 19928 raft_consensus.cc:348] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:18.286706 19928 raft_consensus.cc:374] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.286736 19928 raft_consensus.cc:729] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.286871 19933 tablet_bootstrap.cc:492] T c7376645e76b4c5abdcd721bea1dae5d P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:18.286863 19928 consensus_queue.cc:260] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:18.286892 19925 raft_consensus.cc:348] T 08f66fc67af845b08eaf00717e75d667 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:18.286995 19925 raft_consensus.cc:374] T 08f66fc67af845b08eaf00717e75d667 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.287026 19925 raft_consensus.cc:729] T 08f66fc67af845b08eaf00717e75d667 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.287179 19928 ts_tablet_manager.cc:1398] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:18.287142 19925 consensus_queue.cc:260] T 08f66fc67af845b08eaf00717e75d667 P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:18.287416 19925 ts_tablet_manager.cc:1398] T 08f66fc67af845b08eaf00717e75d667 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:18.287622 19929 tablet_bootstrap.cc:654] T c7376645e76b4c5abdcd721bea1dae5d P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.287726 19933 tablet_bootstrap.cc:654] T c7376645e76b4c5abdcd721bea1dae5d P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.288690 19929 tablet_bootstrap.cc:492] T c7376645e76b4c5abdcd721bea1dae5d P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:18.288769 19929 ts_tablet_manager.cc:1367] T c7376645e76b4c5abdcd721bea1dae5d P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 07:58:18.288830 19933 tablet_bootstrap.cc:492] T c7376645e76b4c5abdcd721bea1dae5d P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:18.288904 19933 ts_tablet_manager.cc:1367] T c7376645e76b4c5abdcd721bea1dae5d P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:58:18.289181 19929 raft_consensus.cc:348] T c7376645e76b4c5abdcd721bea1dae5d P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:18.289278 19929 raft_consensus.cc:374] T c7376645e76b4c5abdcd721bea1dae5d P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.289306 19929 raft_consensus.cc:729] T c7376645e76b4c5abdcd721bea1dae5d P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.289283 19933 raft_consensus.cc:348] T c7376645e76b4c5abdcd721bea1dae5d P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:18.289382 19933 raft_consensus.cc:374] T c7376645e76b4c5abdcd721bea1dae5d P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.289417 19933 raft_consensus.cc:729] T c7376645e76b4c5abdcd721bea1dae5d P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.289433 19929 consensus_queue.cc:260] T c7376645e76b4c5abdcd721bea1dae5d P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:18.289542 19933 consensus_queue.cc:260] T c7376645e76b4c5abdcd721bea1dae5d P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:18.289772 19929 ts_tablet_manager.cc:1398] T c7376645e76b4c5abdcd721bea1dae5d P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:18.289851 19933 ts_tablet_manager.cc:1398] T c7376645e76b4c5abdcd721bea1dae5d P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:18.291400 19178 tablet_service.cc:1467] Processing CreateTablet for tablet 84aed0b3a5e74500b8646d0dbe9b26a1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d38382222f14478fb52d57185405af94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.291649 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84aed0b3a5e74500b8646d0dbe9b26a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.293426 19933 tablet_bootstrap.cc:492] T 84aed0b3a5e74500b8646d0dbe9b26a1 P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:18.293965 18915 tablet_service.cc:1467] Processing CreateTablet for tablet 84aed0b3a5e74500b8646d0dbe9b26a1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=d38382222f14478fb52d57185405af94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.294250 18915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84aed0b3a5e74500b8646d0dbe9b26a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.294395 19933 tablet_bootstrap.cc:654] T 84aed0b3a5e74500b8646d0dbe9b26a1 P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.295907 19933 tablet_bootstrap.cc:492] T 84aed0b3a5e74500b8646d0dbe9b26a1 P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:18.296012 19933 ts_tablet_manager.cc:1367] T 84aed0b3a5e74500b8646d0dbe9b26a1 P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 07:58:18.296185 19926 tablet_bootstrap.cc:492] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:18.296376 18914 tablet_service.cc:1467] Processing CreateTablet for tablet c7376645e76b4c5abdcd721bea1dae5d (DEFAULT_TABLE table=rebalance_test_table_8 [id=d38382222f14478fb52d57185405af94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.296595 18914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7376645e76b4c5abdcd721bea1dae5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.296829 19933 raft_consensus.cc:348] T 84aed0b3a5e74500b8646d0dbe9b26a1 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.296938 19933 raft_consensus.cc:374] T 84aed0b3a5e74500b8646d0dbe9b26a1 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.296972 19933 raft_consensus.cc:729] T 84aed0b3a5e74500b8646d0dbe9b26a1 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.297049 19926 tablet_bootstrap.cc:654] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.297318 19933 consensus_queue.cc:260] T 84aed0b3a5e74500b8646d0dbe9b26a1 P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.298343 19933 ts_tablet_manager.cc:1398] T 84aed0b3a5e74500b8646d0dbe9b26a1 P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:18.299005 19926 tablet_bootstrap.cc:492] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:18.299082 19926 ts_tablet_manager.cc:1367] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:18.299542 19926 raft_consensus.cc:348] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.299649 19926 raft_consensus.cc:374] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.299686 19926 raft_consensus.cc:729] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.299865 19926 consensus_queue.cc:260] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.300163 19926 ts_tablet_manager.cc:1398] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:18.300364 19926 tablet_bootstrap.cc:492] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:18.301738 19926 tablet_bootstrap.cc:654] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.302706 19926 tablet_bootstrap.cc:492] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:18.302783 19926 ts_tablet_manager.cc:1367] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:58:18.303110 19926 raft_consensus.cc:348] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:18.303192 19926 raft_consensus.cc:374] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.303225 19926 raft_consensus.cc:729] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.303349 19926 consensus_queue.cc:260] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:18.303606 19926 ts_tablet_manager.cc:1398] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:18.318254 19865 raft_consensus.cc:482] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:18.318394 19865 raft_consensus.cc:504] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.318799 19865 leader_election.cc:288] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389)
I20240502 07:58:18.319178 18934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84aed0b3a5e74500b8646d0dbe9b26a1" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5" is_pre_election: true
I20240502 07:58:18.319185 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84aed0b3a5e74500b8646d0dbe9b26a1" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662" is_pre_election: true
I20240502 07:58:18.319310 18934 raft_consensus.cc:2454] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 0.
I20240502 07:58:18.319324 19199 raft_consensus.cc:2454] T 84aed0b3a5e74500b8646d0dbe9b26a1 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 0.
I20240502 07:58:18.319617 18605 leader_election.cc:302] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, 9f21202c69df478796a5d7dcfe8b07b5; no voters:
I20240502 07:58:18.319772 19865 raft_consensus.cc:2781] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:18.319847 19865 raft_consensus.cc:482] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:18.319901 19865 raft_consensus.cc:3037] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.343101 19666 raft_consensus.cc:482] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:18.343240 19666 raft_consensus.cc:504] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:18.343647 19666 leader_election.cc:288] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999), 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417)
I20240502 07:58:18.344063 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f66fc67af845b08eaf00717e75d667" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" is_pre_election: true
I20240502 07:58:18.344172 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f66fc67af845b08eaf00717e75d667" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260" is_pre_election: true
I20240502 07:58:18.344236 19066 raft_consensus.cc:2454] T 08f66fc67af845b08eaf00717e75d667 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 0.
I20240502 07:58:18.344317 18803 raft_consensus.cc:2454] T 08f66fc67af845b08eaf00717e75d667 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 0.
I20240502 07:58:18.344678 19264 leader_election.cc:302] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345ab146afdf4d39a1d8583881c06743, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:18.344831 19666 raft_consensus.cc:2781] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:18.344884 19666 raft_consensus.cc:482] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:18.344926 19666 raft_consensus.cc:3037] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.362452 19764 raft_consensus.cc:482] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:18.362589 19764 raft_consensus.cc:504] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:18.362967 19764 leader_election.cc:288] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:18.363335 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7376645e76b4c5abdcd721bea1dae5d" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662" is_pre_election: true
I20240502 07:58:18.363335 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7376645e76b4c5abdcd721bea1dae5d" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a" is_pre_election: true
I20240502 07:58:18.363478 19199 raft_consensus.cc:2454] T c7376645e76b4c5abdcd721bea1dae5d P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:18.363492 18670 raft_consensus.cc:2454] T c7376645e76b4c5abdcd721bea1dae5d P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:18.363781 18870 leader_election.cc:302] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, 9f21202c69df478796a5d7dcfe8b07b5; no voters:
I20240502 07:58:18.363992 19764 raft_consensus.cc:2781] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:18.364058 19764 raft_consensus.cc:482] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:18.364087 19764 raft_consensus.cc:3037] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.365806 19666 raft_consensus.cc:504] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:18.365804 19865 raft_consensus.cc:504] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.365801 19764 raft_consensus.cc:504] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:18.366184 19865 leader_election.cc:288] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 election: Requested vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389)
I20240502 07:58:18.366183 19666 leader_election.cc:288] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 election: Requested vote from peers 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999), 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417)
I20240502 07:58:18.366223 19764 leader_election.cc:288] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Requested vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:18.366606 18934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84aed0b3a5e74500b8646d0dbe9b26a1" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5"
I20240502 07:58:18.366712 18934 raft_consensus.cc:3037] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.366955 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f66fc67af845b08eaf00717e75d667" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3"
I20240502 07:58:18.367072 19066 raft_consensus.cc:3037] T 08f66fc67af845b08eaf00717e75d667 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.367033 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7376645e76b4c5abdcd721bea1dae5d" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a"
I20240502 07:58:18.367144 18670 raft_consensus.cc:3037] T c7376645e76b4c5abdcd721bea1dae5d P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.367349 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08f66fc67af845b08eaf00717e75d667" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260"
I20240502 07:58:18.367465 18803 raft_consensus.cc:3037] T 08f66fc67af845b08eaf00717e75d667 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.367637 18934 raft_consensus.cc:2454] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 1.
I20240502 07:58:18.367985 18605 leader_election.cc:302] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, 9f21202c69df478796a5d7dcfe8b07b5; no voters:
I20240502 07:58:18.368137 18670 raft_consensus.cc:2454] T c7376645e76b4c5abdcd721bea1dae5d P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
I20240502 07:58:18.368172 19865 raft_consensus.cc:2781] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:18.368168 19066 raft_consensus.cc:2454] T 08f66fc67af845b08eaf00717e75d667 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 1.
I20240502 07:58:18.368242 19865 raft_consensus.cc:686] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [term 1 LEADER]: Becoming Leader. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Running, Role: LEADER
I20240502 07:58:18.368376 19865 consensus_queue.cc:237] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.368557 19266 leader_election.cc:302] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 345ab146afdf4d39a1d8583881c06743; no voters:
I20240502 07:58:18.368584 18803 raft_consensus.cc:2454] T 08f66fc67af845b08eaf00717e75d667 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 1.
I20240502 07:58:18.368955 19976 raft_consensus.cc:2781] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:18.369032 19976 raft_consensus.cc:686] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [term 1 LEADER]: Becoming Leader. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Running, Role: LEADER
I20240502 07:58:18.369148 19976 consensus_queue.cc:237] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:18.369249 19199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84aed0b3a5e74500b8646d0dbe9b26a1" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662"
I20240502 07:58:18.369352 19199 raft_consensus.cc:3037] T 84aed0b3a5e74500b8646d0dbe9b26a1 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.369251 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7376645e76b4c5abdcd721bea1dae5d" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662"
I20240502 07:58:18.369868 19198 raft_consensus.cc:3037] T c7376645e76b4c5abdcd721bea1dae5d P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.369896 18870 leader_election.cc:302] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, 9f21202c69df478796a5d7dcfe8b07b5; no voters:
I20240502 07:58:18.370345 19199 raft_consensus.cc:2454] T 84aed0b3a5e74500b8646d0dbe9b26a1 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 1.
I20240502 07:58:18.370642 19764 raft_consensus.cc:2781] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:18.370748 19764 raft_consensus.cc:686] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 LEADER]: Becoming Leader. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Running, Role: LEADER
I20240502 07:58:18.370875 19764 consensus_queue.cc:237] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:18.371291 18523 catalog_manager.cc:5484] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f6bc76c600c413d9f72526563d3234a (127.18.13.129). New cstate: current_term: 1 leader_uuid: "2f6bc76c600c413d9f72526563d3234a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:18.371372 18520 catalog_manager.cc:5484] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 reported cstate change: term changed from 0 to 1, leader changed from <none> to 345ab146afdf4d39a1d8583881c06743 (127.18.13.134). New cstate: current_term: 1 leader_uuid: "345ab146afdf4d39a1d8583881c06743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:18.372776 18526 catalog_manager.cc:5484] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131). New cstate: current_term: 1 leader_uuid: "9f21202c69df478796a5d7dcfe8b07b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:18.388599 19198 raft_consensus.cc:2454] T c7376645e76b4c5abdcd721bea1dae5d P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
W20240502 07:58:18.501224 19999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:18.501441 19999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:18.504416 19999 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 07:58:18.584851 18934 raft_consensus.cc:1261] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Refusing update from remote peer 2f6bc76c600c413d9f72526563d3234a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.584851 19198 raft_consensus.cc:1261] T 84aed0b3a5e74500b8646d0dbe9b26a1 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Refusing update from remote peer 2f6bc76c600c413d9f72526563d3234a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.585294 19804 consensus_queue.cc:1035] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:18.585742 19865 consensus_queue.cc:1035] T 84aed0b3a5e74500b8646d0dbe9b26a1 P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:18.589954 19066 raft_consensus.cc:1261] T 08f66fc67af845b08eaf00717e75d667 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Refusing update from remote peer 345ab146afdf4d39a1d8583881c06743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.590453 19666 consensus_queue.cc:1035] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:18.590590 18803 raft_consensus.cc:1261] T 08f66fc67af845b08eaf00717e75d667 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Refusing update from remote peer 345ab146afdf4d39a1d8583881c06743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.591696 19976 consensus_queue.cc:1035] T 08f66fc67af845b08eaf00717e75d667 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:18.608054 19199 raft_consensus.cc:1261] T c7376645e76b4c5abdcd721bea1dae5d P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Refusing update from remote peer 9f21202c69df478796a5d7dcfe8b07b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.608464 18670 raft_consensus.cc:1261] T c7376645e76b4c5abdcd721bea1dae5d P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Refusing update from remote peer 9f21202c69df478796a5d7dcfe8b07b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:18.608990 19764 consensus_queue.cc:1035] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:18.611207 19764 consensus_queue.cc:1035] T c7376645e76b4c5abdcd721bea1dae5d P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:18.613674 19807 mvcc.cc:205] Tried to move back new op lower bound from 7023151917398163456 to 7023151916518309888. Current Snapshot: MvccSnapshot[applied={T|T < 7023151917398163456}]
I20240502 07:58:18.631322 20037 mvcc.cc:205] Tried to move back new op lower bound from 7023151917498372096 to 7023151916528635904. Current Snapshot: MvccSnapshot[applied={T|T < 7023151917485416448}]
INSERT report
rows total: 2000
time total: 109.385 ms
time per row: 0.0546923 ms
I20240502 07:58:18.659417 18528 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45150:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:18.665191 19046 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce2abb8fd5345b1ba544ba04b82e2ea (DEFAULT_TABLE table=rebalance_test_table_9 [id=843b501e4694422eb972438b9bd57eb2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.665191 19310 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce2abb8fd5345b1ba544ba04b82e2ea (DEFAULT_TABLE table=rebalance_test_table_9 [id=843b501e4694422eb972438b9bd57eb2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.665478 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce2abb8fd5345b1ba544ba04b82e2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.665478 19046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce2abb8fd5345b1ba544ba04b82e2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.667546 19925 tablet_bootstrap.cc:492] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:18.667980 18650 tablet_service.cc:1467] Processing CreateTablet for tablet fc354000a8404e7c95a2ca5f5565c2ce (DEFAULT_TABLE table=rebalance_test_table_9 [id=843b501e4694422eb972438b9bd57eb2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.668267 18650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc354000a8404e7c95a2ca5f5565c2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.668442 19925 tablet_bootstrap.cc:654] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.668747 18783 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce2abb8fd5345b1ba544ba04b82e2ea (DEFAULT_TABLE table=rebalance_test_table_9 [id=843b501e4694422eb972438b9bd57eb2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.668984 18783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce2abb8fd5345b1ba544ba04b82e2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.669010 19928 tablet_bootstrap.cc:492] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:18.669800 19928 tablet_bootstrap.cc:654] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.670290 19929 tablet_bootstrap.cc:492] T fc354000a8404e7c95a2ca5f5565c2ce P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:18.671038 19924 tablet_bootstrap.cc:492] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:18.671183 19929 tablet_bootstrap.cc:654] T fc354000a8404e7c95a2ca5f5565c2ce P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.671238 18782 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7eaa100f764d898c106360d07363fa (DEFAULT_TABLE table=rebalance_test_table_9 [id=843b501e4694422eb972438b9bd57eb2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.671455 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7eaa100f764d898c106360d07363fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.672847 19928 tablet_bootstrap.cc:492] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:18.672924 19928 ts_tablet_manager.cc:1367] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 07:58:18.673292 19928 raft_consensus.cc:348] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.673292 19178 tablet_service.cc:1467] Processing CreateTablet for tablet fc354000a8404e7c95a2ca5f5565c2ce (DEFAULT_TABLE table=rebalance_test_table_9 [id=843b501e4694422eb972438b9bd57eb2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.673395 19928 raft_consensus.cc:374] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.673429 19928 raft_consensus.cc:729] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.673554 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc354000a8404e7c95a2ca5f5565c2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.673545 19928 consensus_queue.cc:260] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.673772 19928 ts_tablet_manager.cc:1398] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:18.675691 19933 tablet_bootstrap.cc:492] T fc354000a8404e7c95a2ca5f5565c2ce P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:18.676614 19933 tablet_bootstrap.cc:654] T fc354000a8404e7c95a2ca5f5565c2ce P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.677659 19310 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7eaa100f764d898c106360d07363fa (DEFAULT_TABLE table=rebalance_test_table_9 [id=843b501e4694422eb972438b9bd57eb2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.677893 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7eaa100f764d898c106360d07363fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.677940 18914 tablet_service.cc:1467] Processing CreateTablet for tablet fc354000a8404e7c95a2ca5f5565c2ce (DEFAULT_TABLE table=rebalance_test_table_9 [id=843b501e4694422eb972438b9bd57eb2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.678179 18914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc354000a8404e7c95a2ca5f5565c2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.678326 19933 tablet_bootstrap.cc:492] T fc354000a8404e7c95a2ca5f5565c2ce P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:18.678383 19933 ts_tablet_manager.cc:1367] T fc354000a8404e7c95a2ca5f5565c2ce P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:58:18.678679 19933 raft_consensus.cc:348] T fc354000a8404e7c95a2ca5f5565c2ce P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.678748 19933 raft_consensus.cc:374] T fc354000a8404e7c95a2ca5f5565c2ce P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.678768 19933 raft_consensus.cc:729] T fc354000a8404e7c95a2ca5f5565c2ce P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.678872 19933 consensus_queue.cc:260] T fc354000a8404e7c95a2ca5f5565c2ce P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.679117 19933 ts_tablet_manager.cc:1398] T fc354000a8404e7c95a2ca5f5565c2ce P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:18.679411 19046 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7eaa100f764d898c106360d07363fa (DEFAULT_TABLE table=rebalance_test_table_9 [id=843b501e4694422eb972438b9bd57eb2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:18.679638 19046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7eaa100f764d898c106360d07363fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:18.680234 19926 tablet_bootstrap.cc:492] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:18.680389 19928 tablet_bootstrap.cc:492] T 7d7eaa100f764d898c106360d07363fa P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:18.681097 19926 tablet_bootstrap.cc:654] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.681324 19928 tablet_bootstrap.cc:654] T 7d7eaa100f764d898c106360d07363fa P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.682340 19926 tablet_bootstrap.cc:492] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:18.682448 19926 ts_tablet_manager.cc:1367] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 07:58:18.682854 19929 tablet_bootstrap.cc:492] T fc354000a8404e7c95a2ca5f5565c2ce P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:18.682945 19929 ts_tablet_manager.cc:1367] T fc354000a8404e7c95a2ca5f5565c2ce P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.002s
I20240502 07:58:18.682967 19926 raft_consensus.cc:348] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.683059 19926 raft_consensus.cc:374] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.683080 19926 raft_consensus.cc:729] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.683189 19926 consensus_queue.cc:260] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.683243 19928 tablet_bootstrap.cc:492] T 7d7eaa100f764d898c106360d07363fa P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:18.683310 19928 ts_tablet_manager.cc:1367] T 7d7eaa100f764d898c106360d07363fa P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240502 07:58:18.683432 19926 ts_tablet_manager.cc:1398] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:18.683414 19929 raft_consensus.cc:348] T fc354000a8404e7c95a2ca5f5565c2ce P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.683517 19929 raft_consensus.cc:374] T fc354000a8404e7c95a2ca5f5565c2ce P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.683549 19929 raft_consensus.cc:729] T fc354000a8404e7c95a2ca5f5565c2ce P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.683689 19929 consensus_queue.cc:260] T fc354000a8404e7c95a2ca5f5565c2ce P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.683976 19929 ts_tablet_manager.cc:1398] T fc354000a8404e7c95a2ca5f5565c2ce P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:18.684046 19928 raft_consensus.cc:348] T 7d7eaa100f764d898c106360d07363fa P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.684136 19928 raft_consensus.cc:374] T 7d7eaa100f764d898c106360d07363fa P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.684170 19928 raft_consensus.cc:729] T 7d7eaa100f764d898c106360d07363fa P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.684283 19928 consensus_queue.cc:260] T 7d7eaa100f764d898c106360d07363fa P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.684628 19928 ts_tablet_manager.cc:1398] T 7d7eaa100f764d898c106360d07363fa P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:18.685025 19925 tablet_bootstrap.cc:492] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:18.685120 19925 ts_tablet_manager.cc:1367] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240502 07:58:18.685689 19925 raft_consensus.cc:348] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.685815 19925 raft_consensus.cc:374] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.685858 19925 raft_consensus.cc:729] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.685987 19925 consensus_queue.cc:260] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.686264 19925 ts_tablet_manager.cc:1398] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:18.686419 19925 tablet_bootstrap.cc:492] T 7d7eaa100f764d898c106360d07363fa P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:18.687337 19925 tablet_bootstrap.cc:654] T 7d7eaa100f764d898c106360d07363fa P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.688123 19924 tablet_bootstrap.cc:654] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.688290 19940 raft_consensus.cc:482] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:18.688367 19940 raft_consensus.cc:504] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.688716 19940 leader_election.cc:288] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647), 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999)
I20240502 07:58:18.689354 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce2abb8fd5345b1ba544ba04b82e2ea" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743" is_pre_election: true
I20240502 07:58:18.689478 19331 raft_consensus.cc:2454] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 0.
I20240502 07:58:18.689574 19924 tablet_bootstrap.cc:492] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:18.689661 19924 ts_tablet_manager.cc:1367] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240502 07:58:18.689713 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce2abb8fd5345b1ba544ba04b82e2ea" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260" is_pre_election: true
I20240502 07:58:18.690109 19924 raft_consensus.cc:348] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.690203 19924 raft_consensus.cc:374] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.690233 19924 raft_consensus.cc:729] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.690290 19000 leader_election.cc:302] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 345ab146afdf4d39a1d8583881c06743; no voters:
I20240502 07:58:18.690351 19924 consensus_queue.cc:260] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.690582 19824 raft_consensus.cc:2781] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:18.690634 19824 raft_consensus.cc:482] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:18.690654 19924 ts_tablet_manager.cc:1398] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:18.690668 19824 raft_consensus.cc:3037] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.690785 19924 tablet_bootstrap.cc:492] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:18.691604 19924 tablet_bootstrap.cc:654] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:18.692662 18803 raft_consensus.cc:2454] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 0.
I20240502 07:58:18.693147 19924 tablet_bootstrap.cc:492] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:18.693255 19924 ts_tablet_manager.cc:1367] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:18.693665 19924 raft_consensus.cc:348] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.693745 19924 raft_consensus.cc:374] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.693778 19924 raft_consensus.cc:729] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.693902 19924 consensus_queue.cc:260] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.694167 19924 ts_tablet_manager.cc:1398] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:18.704216 19925 tablet_bootstrap.cc:492] T 7d7eaa100f764d898c106360d07363fa P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:18.704342 19925 ts_tablet_manager.cc:1367] T 7d7eaa100f764d898c106360d07363fa P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240502 07:58:18.704752 19824 raft_consensus.cc:504] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.704772 19925 raft_consensus.cc:348] T 7d7eaa100f764d898c106360d07363fa P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.704890 19925 raft_consensus.cc:374] T 7d7eaa100f764d898c106360d07363fa P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:18.704938 19925 raft_consensus.cc:729] T 7d7eaa100f764d898c106360d07363fa P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:18.705071 19925 consensus_queue.cc:260] T 7d7eaa100f764d898c106360d07363fa P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.705166 19824 leader_election.cc:288] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 election: Requested vote from peers 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647), 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999)
I20240502 07:58:18.705353 19925 ts_tablet_manager.cc:1398] T 7d7eaa100f764d898c106360d07363fa P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:18.705518 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce2abb8fd5345b1ba544ba04b82e2ea" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743"
I20240502 07:58:18.705679 19331 raft_consensus.cc:3037] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.705708 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce2abb8fd5345b1ba544ba04b82e2ea" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260"
I20240502 07:58:18.705824 18803 raft_consensus.cc:3037] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.706737 19331 raft_consensus.cc:2454] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 1.
I20240502 07:58:18.706770 18803 raft_consensus.cc:2454] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 1.
I20240502 07:58:18.707095 19000 leader_election.cc:302] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 345ab146afdf4d39a1d8583881c06743; no voters:
I20240502 07:58:18.707268 19940 raft_consensus.cc:2781] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:18.707353 19940 raft_consensus.cc:686] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 LEADER]: Becoming Leader. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Running, Role: LEADER
I20240502 07:58:18.707525 19940 consensus_queue.cc:237] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.709582 18528 catalog_manager.cc:5484] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132). New cstate: current_term: 1 leader_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:18.780998 19806 raft_consensus.cc:482] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:18.781113 19806 raft_consensus.cc:504] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.781441 19806 leader_election.cc:288] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523), d19f943e214647a3850f51a324017662 (127.18.13.133:40945)
I20240502 07:58:18.781814 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc354000a8404e7c95a2ca5f5565c2ce" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a" is_pre_election: true
I20240502 07:58:18.781816 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc354000a8404e7c95a2ca5f5565c2ce" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662" is_pre_election: true
I20240502 07:58:18.781935 18670 raft_consensus.cc:2454] T fc354000a8404e7c95a2ca5f5565c2ce P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:18.781955 19198 raft_consensus.cc:2454] T fc354000a8404e7c95a2ca5f5565c2ce P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:18.782197 18870 leader_election.cc:302] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, 9f21202c69df478796a5d7dcfe8b07b5; no voters:
I20240502 07:58:18.782411 19806 raft_consensus.cc:2781] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:18.782457 19806 raft_consensus.cc:482] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:18.782500 19806 raft_consensus.cc:3037] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.783282 19806 raft_consensus.cc:504] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.783589 19806 leader_election.cc:288] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Requested vote from peers 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523), d19f943e214647a3850f51a324017662 (127.18.13.133:40945)
I20240502 07:58:18.783901 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc354000a8404e7c95a2ca5f5565c2ce" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a"
I20240502 07:58:18.783994 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc354000a8404e7c95a2ca5f5565c2ce" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662"
I20240502 07:58:18.784041 18670 raft_consensus.cc:3037] T fc354000a8404e7c95a2ca5f5565c2ce P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.784132 19198 raft_consensus.cc:3037] T fc354000a8404e7c95a2ca5f5565c2ce P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.784759 18670 raft_consensus.cc:2454] T fc354000a8404e7c95a2ca5f5565c2ce P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
I20240502 07:58:18.784863 19198 raft_consensus.cc:2454] T fc354000a8404e7c95a2ca5f5565c2ce P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
I20240502 07:58:18.785050 18870 leader_election.cc:302] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, 9f21202c69df478796a5d7dcfe8b07b5; no voters:
I20240502 07:58:18.785197 19806 raft_consensus.cc:2781] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:18.785279 19806 raft_consensus.cc:686] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 LEADER]: Becoming Leader. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Running, Role: LEADER
I20240502 07:58:18.785424 19806 consensus_queue.cc:237] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } }
I20240502 07:58:18.787086 18528 catalog_manager.cc:5484] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131). New cstate: current_term: 1 leader_uuid: "9f21202c69df478796a5d7dcfe8b07b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:18.907006 19983 raft_consensus.cc:482] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:18.907130 19983 raft_consensus.cc:504] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.907550 19983 leader_election.cc:288] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647), 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417)
I20240502 07:58:18.907905 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7eaa100f764d898c106360d07363fa" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" is_pre_election: true
I20240502 07:58:18.907912 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7eaa100f764d898c106360d07363fa" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743" is_pre_election: true
I20240502 07:58:18.908046 19066 raft_consensus.cc:2454] T 7d7eaa100f764d898c106360d07363fa P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 0.
I20240502 07:58:18.908051 19331 raft_consensus.cc:2454] T 7d7eaa100f764d898c106360d07363fa P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 0.
I20240502 07:58:18.908336 18738 leader_election.cc:302] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:18.908509 19983 raft_consensus.cc:2781] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:18.908566 19983 raft_consensus.cc:482] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:18.908609 19983 raft_consensus.cc:3037] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.909452 19983 raft_consensus.cc:504] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.909766 19983 leader_election.cc:288] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 election: Requested vote from peers 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647), 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417)
I20240502 07:58:18.910076 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7eaa100f764d898c106360d07363fa" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743"
I20240502 07:58:18.910099 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7eaa100f764d898c106360d07363fa" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3"
I20240502 07:58:18.910172 19331 raft_consensus.cc:3037] T 7d7eaa100f764d898c106360d07363fa P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.910185 19066 raft_consensus.cc:3037] T 7d7eaa100f764d898c106360d07363fa P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:18.910898 19331 raft_consensus.cc:2454] T 7d7eaa100f764d898c106360d07363fa P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 1.
I20240502 07:58:18.910959 19066 raft_consensus.cc:2454] T 7d7eaa100f764d898c106360d07363fa P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 1.
I20240502 07:58:18.911132 18736 leader_election.cc:302] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345ab146afdf4d39a1d8583881c06743, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:18.911285 19983 raft_consensus.cc:2781] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:18.911446 19983 raft_consensus.cc:686] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 LEADER]: Becoming Leader. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Running, Role: LEADER
I20240502 07:58:18.911590 19983 consensus_queue.cc:237] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:18.913374 18528 catalog_manager.cc:5484] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "4135e6d90aec4830b08bcb7e8dc50260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: HEALTHY } } }
W20240502 07:58:19.023916 20057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:19.024161 20057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:19.027209 20057 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 07:58:19.123564 18803 raft_consensus.cc:1261] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Refusing update from remote peer 341d0eb74c3544faaa2633d7aa3f77b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:19.123993 19331 raft_consensus.cc:1261] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Refusing update from remote peer 341d0eb74c3544faaa2633d7aa3f77b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:19.124363 19824 consensus_queue.cc:1035] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:19.124541 19940 consensus_queue.cc:1035] T 7ce2abb8fd5345b1ba544ba04b82e2ea P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:19.126652 19066 raft_consensus.cc:1261] T 7d7eaa100f764d898c106360d07363fa P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Refusing update from remote peer 4135e6d90aec4830b08bcb7e8dc50260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:19.127003 19198 raft_consensus.cc:1261] T fc354000a8404e7c95a2ca5f5565c2ce P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Refusing update from remote peer 9f21202c69df478796a5d7dcfe8b07b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:19.127256 18670 raft_consensus.cc:1261] T fc354000a8404e7c95a2ca5f5565c2ce P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Refusing update from remote peer 9f21202c69df478796a5d7dcfe8b07b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:19.128511 19983 consensus_queue.cc:1035] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:19.131314 20043 consensus_queue.cc:1035] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:19.131656 20043 consensus_queue.cc:1035] T fc354000a8404e7c95a2ca5f5565c2ce P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:19.133913 19330 raft_consensus.cc:1261] T 7d7eaa100f764d898c106360d07363fa P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Refusing update from remote peer 4135e6d90aec4830b08bcb7e8dc50260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:19.134408 19983 consensus_queue.cc:1035] T 7d7eaa100f764d898c106360d07363fa P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:19.141242 19730 mvcc.cc:205] Tried to move back new op lower bound from 7023151919606628352 to 7023151917907435520. Current Snapshot: MvccSnapshot[applied={T|T < 7023151919606628352}]
I20240502 07:58:19.150367 19915 mvcc.cc:205] Tried to move back new op lower bound from 7023151919617372160 to 7023151918226411520. Current Snapshot: MvccSnapshot[applied={T|T < 7023151919617372160}]
I20240502 07:58:19.151964 20101 mvcc.cc:205] Tried to move back new op lower bound from 7023151919606628352 to 7023151917907435520. Current Snapshot: MvccSnapshot[applied={T|T < 7023151919606628352}]
I20240502 07:58:19.164685 20091 mvcc.cc:205] Tried to move back new op lower bound from 7023151919617155072 to 7023151918743347200. Current Snapshot: MvccSnapshot[applied={T|T < 7023151919617155072}]
INSERT report
rows total: 2000
time total: 116.868 ms
time per row: 0.0584341 ms
I20240502 07:58:19.197201 18520 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45150:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:19.202697 18914 tablet_service.cc:1467] Processing CreateTablet for tablet 07fd48689c0940909626a09af99e954a (DEFAULT_TABLE table=rebalance_test_table_10 [id=dff6effdb0bf4ea68fc1fe8012dee051]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.202770 19178 tablet_service.cc:1467] Processing CreateTablet for tablet 07fd48689c0940909626a09af99e954a (DEFAULT_TABLE table=rebalance_test_table_10 [id=dff6effdb0bf4ea68fc1fe8012dee051]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.203020 18914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07fd48689c0940909626a09af99e954a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.203059 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07fd48689c0940909626a09af99e954a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.203545 19179 tablet_service.cc:1467] Processing CreateTablet for tablet 97761e6b14b047f1a598750d405063db (DEFAULT_TABLE table=rebalance_test_table_10 [id=dff6effdb0bf4ea68fc1fe8012dee051]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.203567 18915 tablet_service.cc:1467] Processing CreateTablet for tablet 97761e6b14b047f1a598750d405063db (DEFAULT_TABLE table=rebalance_test_table_10 [id=dff6effdb0bf4ea68fc1fe8012dee051]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.203747 19179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97761e6b14b047f1a598750d405063db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.204238 19310 tablet_service.cc:1467] Processing CreateTablet for tablet 58e1075a0a4f481c81181f7120dcb798 (DEFAULT_TABLE table=rebalance_test_table_10 [id=dff6effdb0bf4ea68fc1fe8012dee051]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.204494 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e1075a0a4f481c81181f7120dcb798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.205875 19046 tablet_service.cc:1467] Processing CreateTablet for tablet 58e1075a0a4f481c81181f7120dcb798 (DEFAULT_TABLE table=rebalance_test_table_10 [id=dff6effdb0bf4ea68fc1fe8012dee051]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.205906 18650 tablet_service.cc:1467] Processing CreateTablet for tablet 07fd48689c0940909626a09af99e954a (DEFAULT_TABLE table=rebalance_test_table_10 [id=dff6effdb0bf4ea68fc1fe8012dee051]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.206028 20115 tablet_bootstrap.cc:492] T 07fd48689c0940909626a09af99e954a P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:19.206130 19046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e1075a0a4f481c81181f7120dcb798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.206176 18650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07fd48689c0940909626a09af99e954a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.207149 20115 tablet_bootstrap.cc:654] T 07fd48689c0940909626a09af99e954a P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.203768 18915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97761e6b14b047f1a598750d405063db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.208629 18651 tablet_service.cc:1467] Processing CreateTablet for tablet 97761e6b14b047f1a598750d405063db (DEFAULT_TABLE table=rebalance_test_table_10 [id=dff6effdb0bf4ea68fc1fe8012dee051]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.208628 18782 tablet_service.cc:1467] Processing CreateTablet for tablet 58e1075a0a4f481c81181f7120dcb798 (DEFAULT_TABLE table=rebalance_test_table_10 [id=dff6effdb0bf4ea68fc1fe8012dee051]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.208843 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97761e6b14b047f1a598750d405063db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.208997 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e1075a0a4f481c81181f7120dcb798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.210197 20119 tablet_bootstrap.cc:492] T 07fd48689c0940909626a09af99e954a P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:19.211279 20120 tablet_bootstrap.cc:492] T 58e1075a0a4f481c81181f7120dcb798 P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:19.211347 20116 tablet_bootstrap.cc:492] T 58e1075a0a4f481c81181f7120dcb798 P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:19.211475 20118 tablet_bootstrap.cc:492] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:19.212373 20116 tablet_bootstrap.cc:654] T 58e1075a0a4f481c81181f7120dcb798 P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.212451 20118 tablet_bootstrap.cc:654] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.212517 20119 tablet_bootstrap.cc:654] T 07fd48689c0940909626a09af99e954a P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.212815 20120 tablet_bootstrap.cc:654] T 58e1075a0a4f481c81181f7120dcb798 P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.213761 20115 tablet_bootstrap.cc:492] T 07fd48689c0940909626a09af99e954a P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:19.213867 20115 ts_tablet_manager.cc:1367] T 07fd48689c0940909626a09af99e954a P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 07:58:19.213878 20119 tablet_bootstrap.cc:492] T 07fd48689c0940909626a09af99e954a P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:19.213958 20119 ts_tablet_manager.cc:1367] T 07fd48689c0940909626a09af99e954a P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 07:58:19.214210 20118 tablet_bootstrap.cc:492] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:19.214298 20118 ts_tablet_manager.cc:1367] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:19.214342 20115 raft_consensus.cc:348] T 07fd48689c0940909626a09af99e954a P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:19.214392 20119 raft_consensus.cc:348] T 07fd48689c0940909626a09af99e954a P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:19.214462 20115 raft_consensus.cc:374] T 07fd48689c0940909626a09af99e954a P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.214499 20115 raft_consensus.cc:729] T 07fd48689c0940909626a09af99e954a P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.214529 20119 raft_consensus.cc:374] T 07fd48689c0940909626a09af99e954a P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.214576 20119 raft_consensus.cc:729] T 07fd48689c0940909626a09af99e954a P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.214666 20115 consensus_queue.cc:260] T 07fd48689c0940909626a09af99e954a P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:19.214701 20119 consensus_queue.cc:260] T 07fd48689c0940909626a09af99e954a P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:19.214766 20118 raft_consensus.cc:348] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.214955 20115 ts_tablet_manager.cc:1398] T 07fd48689c0940909626a09af99e954a P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:19.215178 20115 tablet_bootstrap.cc:492] T 97761e6b14b047f1a598750d405063db P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:19.215348 20118 raft_consensus.cc:374] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.215454 20118 raft_consensus.cc:729] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.215646 20118 consensus_queue.cc:260] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.216246 20119 ts_tablet_manager.cc:1398] T 07fd48689c0940909626a09af99e954a P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:19.216410 20119 tablet_bootstrap.cc:492] T 97761e6b14b047f1a598750d405063db P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:19.216648 20118 ts_tablet_manager.cc:1398] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:19.217337 20120 tablet_bootstrap.cc:492] T 58e1075a0a4f481c81181f7120dcb798 P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:19.217947 20120 ts_tablet_manager.cc:1367] T 58e1075a0a4f481c81181f7120dcb798 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 07:58:19.218150 20115 tablet_bootstrap.cc:654] T 97761e6b14b047f1a598750d405063db P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.218943 20120 raft_consensus.cc:348] T 58e1075a0a4f481c81181f7120dcb798 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.219071 20120 raft_consensus.cc:374] T 58e1075a0a4f481c81181f7120dcb798 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.219148 20120 raft_consensus.cc:729] T 58e1075a0a4f481c81181f7120dcb798 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.219399 20120 consensus_queue.cc:260] T 58e1075a0a4f481c81181f7120dcb798 P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.219573 20115 tablet_bootstrap.cc:492] T 97761e6b14b047f1a598750d405063db P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:19.219672 20115 ts_tablet_manager.cc:1367] T 97761e6b14b047f1a598750d405063db P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:58:19.220288 20115 raft_consensus.cc:348] T 97761e6b14b047f1a598750d405063db P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:19.220453 20120 ts_tablet_manager.cc:1398] T 58e1075a0a4f481c81181f7120dcb798 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:58:19.220461 20115 raft_consensus.cc:374] T 97761e6b14b047f1a598750d405063db P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.220513 20115 raft_consensus.cc:729] T 97761e6b14b047f1a598750d405063db P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.220681 20115 consensus_queue.cc:260] T 97761e6b14b047f1a598750d405063db P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:19.220846 20119 tablet_bootstrap.cc:654] T 97761e6b14b047f1a598750d405063db P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.221097 20115 ts_tablet_manager.cc:1398] T 97761e6b14b047f1a598750d405063db P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:19.221526 20114 tablet_bootstrap.cc:492] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:19.222424 20114 tablet_bootstrap.cc:654] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.223455 20114 tablet_bootstrap.cc:492] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:19.223551 20114 ts_tablet_manager.cc:1367] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:19.224036 20114 raft_consensus.cc:348] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:19.224104 20116 tablet_bootstrap.cc:492] T 58e1075a0a4f481c81181f7120dcb798 P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:19.224196 20116 ts_tablet_manager.cc:1367] T 58e1075a0a4f481c81181f7120dcb798 P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 07:58:19.224213 20114 raft_consensus.cc:374] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.224311 20114 raft_consensus.cc:729] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.224493 20114 consensus_queue.cc:260] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:19.224858 20114 ts_tablet_manager.cc:1398] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:19.224823 20116 raft_consensus.cc:348] T 58e1075a0a4f481c81181f7120dcb798 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.224928 20116 raft_consensus.cc:374] T 58e1075a0a4f481c81181f7120dcb798 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.225000 20116 raft_consensus.cc:729] T 58e1075a0a4f481c81181f7120dcb798 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.225031 20114 tablet_bootstrap.cc:492] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:19.225140 20116 consensus_queue.cc:260] T 58e1075a0a4f481c81181f7120dcb798 P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.225507 20116 ts_tablet_manager.cc:1398] T 58e1075a0a4f481c81181f7120dcb798 P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:19.225888 20114 tablet_bootstrap.cc:654] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.227007 20119 tablet_bootstrap.cc:492] T 97761e6b14b047f1a598750d405063db P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:19.227085 20114 tablet_bootstrap.cc:492] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:19.227093 20119 ts_tablet_manager.cc:1367] T 97761e6b14b047f1a598750d405063db P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 07:58:19.227149 20114 ts_tablet_manager.cc:1367] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:19.227490 20114 raft_consensus.cc:348] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:19.227497 20119 raft_consensus.cc:348] T 97761e6b14b047f1a598750d405063db P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:19.227581 20114 raft_consensus.cc:374] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.227582 20119 raft_consensus.cc:374] T 97761e6b14b047f1a598750d405063db P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.227613 20114 raft_consensus.cc:729] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.227613 20119 raft_consensus.cc:729] T 97761e6b14b047f1a598750d405063db P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.227859 20119 consensus_queue.cc:260] T 97761e6b14b047f1a598750d405063db P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:19.228117 20114 consensus_queue.cc:260] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:19.228189 20119 ts_tablet_manager.cc:1398] T 97761e6b14b047f1a598750d405063db P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:19.229343 20114 ts_tablet_manager.cc:1398] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:19.355901 19764 raft_consensus.cc:482] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:19.356024 19764 raft_consensus.cc:504] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:19.356410 19764 leader_election.cc:288] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:19.356799 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fd48689c0940909626a09af99e954a" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a" is_pre_election: true
I20240502 07:58:19.356796 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fd48689c0940909626a09af99e954a" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662" is_pre_election: true
I20240502 07:58:19.356948 18670 raft_consensus.cc:2454] T 07fd48689c0940909626a09af99e954a P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:19.356986 19198 raft_consensus.cc:2454] T 07fd48689c0940909626a09af99e954a P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:19.357291 18868 leader_election.cc:302] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f21202c69df478796a5d7dcfe8b07b5, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:19.357460 19764 raft_consensus.cc:2781] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:19.357627 19764 raft_consensus.cc:482] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:19.357717 19764 raft_consensus.cc:3037] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:19.358752 19764 raft_consensus.cc:504] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:19.359145 19764 leader_election.cc:288] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Requested vote from peers d19f943e214647a3850f51a324017662 (127.18.13.133:40945), 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523)
I20240502 07:58:19.359409 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fd48689c0940909626a09af99e954a" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662"
I20240502 07:58:19.359506 19198 raft_consensus.cc:3037] T 07fd48689c0940909626a09af99e954a P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:19.359462 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fd48689c0940909626a09af99e954a" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a"
I20240502 07:58:19.359587 18670 raft_consensus.cc:3037] T 07fd48689c0940909626a09af99e954a P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:19.360356 19198 raft_consensus.cc:2454] T 07fd48689c0940909626a09af99e954a P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
I20240502 07:58:19.360464 18670 raft_consensus.cc:2454] T 07fd48689c0940909626a09af99e954a P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
I20240502 07:58:19.360661 18868 leader_election.cc:302] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f21202c69df478796a5d7dcfe8b07b5, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:19.360843 19764 raft_consensus.cc:2781] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:19.360954 19764 raft_consensus.cc:686] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 LEADER]: Becoming Leader. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Running, Role: LEADER
I20240502 07:58:19.361090 19764 consensus_queue.cc:237] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } }
I20240502 07:58:19.363159 18528 catalog_manager.cc:5484] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131). New cstate: current_term: 1 leader_uuid: "9f21202c69df478796a5d7dcfe8b07b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:19.496731 19824 raft_consensus.cc:482] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:19.496876 19824 raft_consensus.cc:504] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.497262 19824 leader_election.cc:288] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647), 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999)
I20240502 07:58:19.497860 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1075a0a4f481c81181f7120dcb798" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260" is_pre_election: true
I20240502 07:58:19.497884 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1075a0a4f481c81181f7120dcb798" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743" is_pre_election: true
I20240502 07:58:19.498011 18803 raft_consensus.cc:2454] T 58e1075a0a4f481c81181f7120dcb798 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 0.
I20240502 07:58:19.498044 19331 raft_consensus.cc:2454] T 58e1075a0a4f481c81181f7120dcb798 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 0.
I20240502 07:58:19.498364 19000 leader_election.cc:302] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:19.498559 19824 raft_consensus.cc:2781] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:19.498615 19824 raft_consensus.cc:482] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:19.498668 19824 raft_consensus.cc:3037] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:19.499553 19824 raft_consensus.cc:504] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.499950 19824 leader_election.cc:288] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 election: Requested vote from peers 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647), 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999)
I20240502 07:58:19.500299 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1075a0a4f481c81181f7120dcb798" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743"
I20240502 07:58:19.500321 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e1075a0a4f481c81181f7120dcb798" candidate_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260"
I20240502 07:58:19.500452 18803 raft_consensus.cc:3037] T 58e1075a0a4f481c81181f7120dcb798 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:19.500442 19331 raft_consensus.cc:3037] T 58e1075a0a4f481c81181f7120dcb798 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:19.501397 19331 raft_consensus.cc:2454] T 58e1075a0a4f481c81181f7120dcb798 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 1.
I20240502 07:58:19.501487 18803 raft_consensus.cc:2454] T 58e1075a0a4f481c81181f7120dcb798 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 341d0eb74c3544faaa2633d7aa3f77b3 in term 1.
I20240502 07:58:19.501675 19000 leader_election.cc:302] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 345ab146afdf4d39a1d8583881c06743; no voters:
I20240502 07:58:19.501873 19824 raft_consensus.cc:2781] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:19.501961 19824 raft_consensus.cc:686] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 LEADER]: Becoming Leader. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Running, Role: LEADER
I20240502 07:58:19.502100 19824 consensus_queue.cc:237] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.503898 18528 catalog_manager.cc:5484] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132). New cstate: current_term: 1 leader_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:19.613034 19806 raft_consensus.cc:482] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:19.613164 19806 raft_consensus.cc:504] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:19.613584 19806 leader_election.cc:288] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523), d19f943e214647a3850f51a324017662 (127.18.13.133:40945)
I20240502 07:58:19.613924 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97761e6b14b047f1a598750d405063db" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a" is_pre_election: true
I20240502 07:58:19.613941 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97761e6b14b047f1a598750d405063db" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662" is_pre_election: true
I20240502 07:58:19.614061 18670 raft_consensus.cc:2454] T 97761e6b14b047f1a598750d405063db P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:19.614082 19198 raft_consensus.cc:2454] T 97761e6b14b047f1a598750d405063db P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 0.
I20240502 07:58:19.614353 18870 leader_election.cc:302] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, 9f21202c69df478796a5d7dcfe8b07b5; no voters:
I20240502 07:58:19.614528 19806 raft_consensus.cc:2781] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:19.614578 19806 raft_consensus.cc:482] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:19.614603 19806 raft_consensus.cc:3037] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:19.615371 19806 raft_consensus.cc:504] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:19.615681 19806 leader_election.cc:288] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Requested vote from peers 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523), d19f943e214647a3850f51a324017662 (127.18.13.133:40945)
I20240502 07:58:19.616003 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97761e6b14b047f1a598750d405063db" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f6bc76c600c413d9f72526563d3234a"
I20240502 07:58:19.616008 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97761e6b14b047f1a598750d405063db" candidate_uuid: "9f21202c69df478796a5d7dcfe8b07b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662"
I20240502 07:58:19.616122 18670 raft_consensus.cc:3037] T 97761e6b14b047f1a598750d405063db P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:19.616132 19198 raft_consensus.cc:3037] T 97761e6b14b047f1a598750d405063db P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:19.616883 19198 raft_consensus.cc:2454] T 97761e6b14b047f1a598750d405063db P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
I20240502 07:58:19.616904 18670 raft_consensus.cc:2454] T 97761e6b14b047f1a598750d405063db P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21202c69df478796a5d7dcfe8b07b5 in term 1.
I20240502 07:58:19.617170 18868 leader_election.cc:302] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f21202c69df478796a5d7dcfe8b07b5, d19f943e214647a3850f51a324017662; no voters:
I20240502 07:58:19.617353 19806 raft_consensus.cc:2781] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:19.617445 19806 raft_consensus.cc:686] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 LEADER]: Becoming Leader. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Running, Role: LEADER
I20240502 07:58:19.617583 19806 consensus_queue.cc:237] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:19.620767 18528 catalog_manager.cc:5484] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131). New cstate: current_term: 1 leader_uuid: "9f21202c69df478796a5d7dcfe8b07b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:19.773995 20132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:19.774255 20132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:19.777454 20132 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:58:19.796209 19806 consensus_queue.cc:1035] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:19.799559 19764 consensus_queue.cc:1035] T 07fd48689c0940909626a09af99e954a P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_10'
I20240502 07:58:19.870991 18803 raft_consensus.cc:1261] T 58e1075a0a4f481c81181f7120dcb798 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Refusing update from remote peer 341d0eb74c3544faaa2633d7aa3f77b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:19.871387 19331 raft_consensus.cc:1261] T 58e1075a0a4f481c81181f7120dcb798 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Refusing update from remote peer 341d0eb74c3544faaa2633d7aa3f77b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:19.871814 19824 consensus_queue.cc:1035] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:19.872910 19824 consensus_queue.cc:1035] T 58e1075a0a4f481c81181f7120dcb798 P 341d0eb74c3544faaa2633d7aa3f77b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:19.907089 19198 raft_consensus.cc:1261] T 97761e6b14b047f1a598750d405063db P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Refusing update from remote peer 9f21202c69df478796a5d7dcfe8b07b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:19.907507 18670 raft_consensus.cc:1261] T 97761e6b14b047f1a598750d405063db P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Refusing update from remote peer 9f21202c69df478796a5d7dcfe8b07b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:19.907967 20043 consensus_queue.cc:1035] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:19.908145 20173 consensus_queue.cc:1035] T 97761e6b14b047f1a598750d405063db P 9f21202c69df478796a5d7dcfe8b07b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.419 ms
time per row: 0.0592097 ms
I20240502 07:58:19.946437 18522 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45150:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:19.952440 19310 tablet_service.cc:1467] Processing CreateTablet for tablet 161328ecf93f48d399060275e63a8df1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=a9450bb02c8b4dbc9e0d8ce47a328299]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.952805 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161328ecf93f48d399060275e63a8df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.953024 19311 tablet_service.cc:1467] Processing CreateTablet for tablet b726b43a39a7440199728ec8d51817d8 (DEFAULT_TABLE table=rebalance_test_table_11 [id=a9450bb02c8b4dbc9e0d8ce47a328299]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.953254 19311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b726b43a39a7440199728ec8d51817d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.953357 18782 tablet_service.cc:1467] Processing CreateTablet for tablet 161328ecf93f48d399060275e63a8df1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=a9450bb02c8b4dbc9e0d8ce47a328299]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.953517 18783 tablet_service.cc:1467] Processing CreateTablet for tablet b726b43a39a7440199728ec8d51817d8 (DEFAULT_TABLE table=rebalance_test_table_11 [id=a9450bb02c8b4dbc9e0d8ce47a328299]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.953626 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161328ecf93f48d399060275e63a8df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.953761 18783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b726b43a39a7440199728ec8d51817d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.955960 20188 tablet_bootstrap.cc:492] T b726b43a39a7440199728ec8d51817d8 P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:19.955806 18915 tablet_service.cc:1467] Processing CreateTablet for tablet 03fb35f752fd4e3d8e9821ee645b88e2 (DEFAULT_TABLE table=rebalance_test_table_11 [id=a9450bb02c8b4dbc9e0d8ce47a328299]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.956209 18915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fb35f752fd4e3d8e9821ee645b88e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.956400 19046 tablet_service.cc:1467] Processing CreateTablet for tablet 161328ecf93f48d399060275e63a8df1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=a9450bb02c8b4dbc9e0d8ce47a328299]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.956400 19047 tablet_service.cc:1467] Processing CreateTablet for tablet b726b43a39a7440199728ec8d51817d8 (DEFAULT_TABLE table=rebalance_test_table_11 [id=a9450bb02c8b4dbc9e0d8ce47a328299]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.956722 19046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161328ecf93f48d399060275e63a8df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.956856 19047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b726b43a39a7440199728ec8d51817d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.957172 20188 tablet_bootstrap.cc:654] T b726b43a39a7440199728ec8d51817d8 P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.958559 20187 tablet_bootstrap.cc:492] T 161328ecf93f48d399060275e63a8df1 P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:19.958804 20190 tablet_bootstrap.cc:492] T 03fb35f752fd4e3d8e9821ee645b88e2 P 9f21202c69df478796a5d7dcfe8b07b5: Bootstrap starting.
I20240502 07:58:19.959245 18651 tablet_service.cc:1467] Processing CreateTablet for tablet 03fb35f752fd4e3d8e9821ee645b88e2 (DEFAULT_TABLE table=rebalance_test_table_11 [id=a9450bb02c8b4dbc9e0d8ce47a328299]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.959432 20187 tablet_bootstrap.cc:654] T 161328ecf93f48d399060275e63a8df1 P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.959437 20188 tablet_bootstrap.cc:492] T b726b43a39a7440199728ec8d51817d8 P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:19.959525 18651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fb35f752fd4e3d8e9821ee645b88e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.959518 20188 ts_tablet_manager.cc:1367] T b726b43a39a7440199728ec8d51817d8 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 07:58:19.960079 20191 tablet_bootstrap.cc:492] T 161328ecf93f48d399060275e63a8df1 P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:19.960047 20188 raft_consensus.cc:348] T b726b43a39a7440199728ec8d51817d8 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.960147 20188 raft_consensus.cc:374] T b726b43a39a7440199728ec8d51817d8 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.960176 20188 raft_consensus.cc:729] T b726b43a39a7440199728ec8d51817d8 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.960289 20188 consensus_queue.cc:260] T b726b43a39a7440199728ec8d51817d8 P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.960579 20188 ts_tablet_manager.cc:1398] T b726b43a39a7440199728ec8d51817d8 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:19.960736 20188 tablet_bootstrap.cc:492] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260: Bootstrap starting.
I20240502 07:58:19.961028 20191 tablet_bootstrap.cc:654] T 161328ecf93f48d399060275e63a8df1 P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.961491 20188 tablet_bootstrap.cc:654] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.962059 20187 tablet_bootstrap.cc:492] T 161328ecf93f48d399060275e63a8df1 P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:19.962158 20187 ts_tablet_manager.cc:1367] T 161328ecf93f48d399060275e63a8df1 P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:19.962512 20187 raft_consensus.cc:348] T 161328ecf93f48d399060275e63a8df1 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:19.962659 20187 raft_consensus.cc:374] T 161328ecf93f48d399060275e63a8df1 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.962718 20187 raft_consensus.cc:729] T 161328ecf93f48d399060275e63a8df1 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.962868 20187 consensus_queue.cc:260] T 161328ecf93f48d399060275e63a8df1 P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:19.963024 20190 tablet_bootstrap.cc:654] T 03fb35f752fd4e3d8e9821ee645b88e2 P 9f21202c69df478796a5d7dcfe8b07b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.963157 20187 ts_tablet_manager.cc:1398] T 161328ecf93f48d399060275e63a8df1 P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:19.963322 20187 tablet_bootstrap.cc:492] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743: Bootstrap starting.
I20240502 07:58:19.964182 20187 tablet_bootstrap.cc:654] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.964411 19179 tablet_service.cc:1467] Processing CreateTablet for tablet 03fb35f752fd4e3d8e9821ee645b88e2 (DEFAULT_TABLE table=rebalance_test_table_11 [id=a9450bb02c8b4dbc9e0d8ce47a328299]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:19.964692 19179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fb35f752fd4e3d8e9821ee645b88e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.965104 20191 tablet_bootstrap.cc:492] T 161328ecf93f48d399060275e63a8df1 P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:19.965198 20191 ts_tablet_manager.cc:1367] T 161328ecf93f48d399060275e63a8df1 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 07:58:19.965641 20191 raft_consensus.cc:348] T 161328ecf93f48d399060275e63a8df1 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:19.965762 20191 raft_consensus.cc:374] T 161328ecf93f48d399060275e63a8df1 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.965823 20191 raft_consensus.cc:729] T 161328ecf93f48d399060275e63a8df1 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.966027 20191 consensus_queue.cc:260] T 161328ecf93f48d399060275e63a8df1 P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:19.966372 20191 ts_tablet_manager.cc:1398] T 161328ecf93f48d399060275e63a8df1 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:19.966428 20194 tablet_bootstrap.cc:492] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a: Bootstrap starting.
I20240502 07:58:19.966538 20191 tablet_bootstrap.cc:492] T b726b43a39a7440199728ec8d51817d8 P 341d0eb74c3544faaa2633d7aa3f77b3: Bootstrap starting.
I20240502 07:58:19.967092 20198 tablet_bootstrap.cc:492] T 03fb35f752fd4e3d8e9821ee645b88e2 P d19f943e214647a3850f51a324017662: Bootstrap starting.
I20240502 07:58:19.967619 20191 tablet_bootstrap.cc:654] T b726b43a39a7440199728ec8d51817d8 P 341d0eb74c3544faaa2633d7aa3f77b3: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.967657 20194 tablet_bootstrap.cc:654] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.968709 20190 tablet_bootstrap.cc:492] T 03fb35f752fd4e3d8e9821ee645b88e2 P 9f21202c69df478796a5d7dcfe8b07b5: No bootstrap required, opened a new log
I20240502 07:58:19.968780 20191 tablet_bootstrap.cc:492] T b726b43a39a7440199728ec8d51817d8 P 341d0eb74c3544faaa2633d7aa3f77b3: No bootstrap required, opened a new log
I20240502 07:58:19.968852 20190 ts_tablet_manager.cc:1367] T 03fb35f752fd4e3d8e9821ee645b88e2 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 07:58:19.968860 20191 ts_tablet_manager.cc:1367] T b726b43a39a7440199728ec8d51817d8 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 07:58:19.969103 20198 tablet_bootstrap.cc:654] T 03fb35f752fd4e3d8e9821ee645b88e2 P d19f943e214647a3850f51a324017662: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:19.969348 20191 raft_consensus.cc:348] T b726b43a39a7440199728ec8d51817d8 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.969453 20191 raft_consensus.cc:374] T b726b43a39a7440199728ec8d51817d8 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.969484 20191 raft_consensus.cc:729] T b726b43a39a7440199728ec8d51817d8 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341d0eb74c3544faaa2633d7aa3f77b3, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.969604 20191 consensus_queue.cc:260] T b726b43a39a7440199728ec8d51817d8 P 341d0eb74c3544faaa2633d7aa3f77b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.969887 20191 ts_tablet_manager.cc:1398] T b726b43a39a7440199728ec8d51817d8 P 341d0eb74c3544faaa2633d7aa3f77b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:19.970510 20198 tablet_bootstrap.cc:492] T 03fb35f752fd4e3d8e9821ee645b88e2 P d19f943e214647a3850f51a324017662: No bootstrap required, opened a new log
I20240502 07:58:19.970518 20190 raft_consensus.cc:348] T 03fb35f752fd4e3d8e9821ee645b88e2 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:19.970590 20198 ts_tablet_manager.cc:1367] T 03fb35f752fd4e3d8e9821ee645b88e2 P d19f943e214647a3850f51a324017662: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:19.970623 20190 raft_consensus.cc:374] T 03fb35f752fd4e3d8e9821ee645b88e2 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.970659 20190 raft_consensus.cc:729] T 03fb35f752fd4e3d8e9821ee645b88e2 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21202c69df478796a5d7dcfe8b07b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.970789 20187 tablet_bootstrap.cc:492] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743: No bootstrap required, opened a new log
I20240502 07:58:19.970867 20187 ts_tablet_manager.cc:1367] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 07:58:19.970893 20190 consensus_queue.cc:260] T 03fb35f752fd4e3d8e9821ee645b88e2 P 9f21202c69df478796a5d7dcfe8b07b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:19.971033 20198 raft_consensus.cc:348] T 03fb35f752fd4e3d8e9821ee645b88e2 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:19.971125 20198 raft_consensus.cc:374] T 03fb35f752fd4e3d8e9821ee645b88e2 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.971155 20198 raft_consensus.cc:729] T 03fb35f752fd4e3d8e9821ee645b88e2 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19f943e214647a3850f51a324017662, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.971174 20190 ts_tablet_manager.cc:1398] T 03fb35f752fd4e3d8e9821ee645b88e2 P 9f21202c69df478796a5d7dcfe8b07b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:19.971278 20198 consensus_queue.cc:260] T 03fb35f752fd4e3d8e9821ee645b88e2 P d19f943e214647a3850f51a324017662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:19.971330 20187 raft_consensus.cc:348] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.971424 20187 raft_consensus.cc:374] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.971453 20187 raft_consensus.cc:729] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.971580 20198 ts_tablet_manager.cc:1398] T 03fb35f752fd4e3d8e9821ee645b88e2 P d19f943e214647a3850f51a324017662: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:19.971561 20187 consensus_queue.cc:260] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.971855 20188 tablet_bootstrap.cc:492] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260: No bootstrap required, opened a new log
I20240502 07:58:19.971882 20202 raft_consensus.cc:482] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:19.971947 20188 ts_tablet_manager.cc:1367] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240502 07:58:19.971942 20187 ts_tablet_manager.cc:1398] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:19.971949 20202 raft_consensus.cc:504] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.972419 20188 raft_consensus.cc:348] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:19.972519 20188 raft_consensus.cc:374] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.972549 20188 raft_consensus.cc:729] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.972664 20188 consensus_queue.cc:260] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:19.972915 20188 ts_tablet_manager.cc:1398] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:19.974119 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b726b43a39a7440199728ec8d51817d8" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" is_pre_election: true
I20240502 07:58:19.974247 19066 raft_consensus.cc:2454] T b726b43a39a7440199728ec8d51817d8 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 0.
I20240502 07:58:19.974418 20194 tablet_bootstrap.cc:492] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a: No bootstrap required, opened a new log
I20240502 07:58:19.974515 20194 ts_tablet_manager.cc:1367] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 07:58:19.974532 19266 leader_election.cc:302] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 345ab146afdf4d39a1d8583881c06743; no voters:
I20240502 07:58:19.974691 20100 raft_consensus.cc:2781] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:19.974735 20100 raft_consensus.cc:482] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:19.974769 20100 raft_consensus.cc:3037] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:19.975015 20194 raft_consensus.cc:348] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:19.975121 20194 raft_consensus.cc:374] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:19.975162 20194 raft_consensus.cc:729] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Initialized, Role: FOLLOWER
I20240502 07:58:19.975275 20194 consensus_queue.cc:260] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:19.975526 20194 ts_tablet_manager.cc:1398] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:19.975858 20100 raft_consensus.cc:504] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.976070 20202 leader_election.cc:288] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417), 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999)
I20240502 07:58:19.976162 20100 leader_election.cc:288] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 election: Requested vote from peers 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417), 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130:43999)
I20240502 07:58:19.976536 18803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b726b43a39a7440199728ec8d51817d8" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260" is_pre_election: true
I20240502 07:58:19.976591 18802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b726b43a39a7440199728ec8d51817d8" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4135e6d90aec4830b08bcb7e8dc50260"
I20240502 07:58:19.976672 18803 raft_consensus.cc:2454] T b726b43a39a7440199728ec8d51817d8 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 0.
I20240502 07:58:19.976855 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b726b43a39a7440199728ec8d51817d8" candidate_uuid: "345ab146afdf4d39a1d8583881c06743" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3"
I20240502 07:58:19.976964 19066 raft_consensus.cc:3037] T b726b43a39a7440199728ec8d51817d8 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:19.977883 19066 raft_consensus.cc:2454] T b726b43a39a7440199728ec8d51817d8 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345ab146afdf4d39a1d8583881c06743 in term 1.
I20240502 07:58:19.978137 19266 leader_election.cc:302] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 345ab146afdf4d39a1d8583881c06743; no voters: 4135e6d90aec4830b08bcb7e8dc50260
I20240502 07:58:19.978277 20100 raft_consensus.cc:2781] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:19.978366 20100 raft_consensus.cc:686] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [term 1 LEADER]: Becoming Leader. State: Replica: 345ab146afdf4d39a1d8583881c06743, State: Running, Role: LEADER
I20240502 07:58:19.978493 20100 consensus_queue.cc:237] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } }
I20240502 07:58:19.980104 18520 catalog_manager.cc:5484] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 reported cstate change: term changed from 0 to 1, leader changed from <none> to 345ab146afdf4d39a1d8583881c06743 (127.18.13.134). New cstate: current_term: 1 leader_uuid: "345ab146afdf4d39a1d8583881c06743" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:20.141276 20039 raft_consensus.cc:482] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:20.141395 20039 raft_consensus.cc:504] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:20.141856 20039 leader_election.cc:288] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389), d19f943e214647a3850f51a324017662 (127.18.13.133:40945)
I20240502 07:58:20.142266 18934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fb35f752fd4e3d8e9821ee645b88e2" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5" is_pre_election: true
I20240502 07:58:20.142267 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fb35f752fd4e3d8e9821ee645b88e2" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662" is_pre_election: true
I20240502 07:58:20.142397 18934 raft_consensus.cc:2454] T 03fb35f752fd4e3d8e9821ee645b88e2 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 0.
I20240502 07:58:20.142408 19198 raft_consensus.cc:2454] T 03fb35f752fd4e3d8e9821ee645b88e2 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 0.
I20240502 07:58:20.142715 18605 leader_election.cc:302] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, 9f21202c69df478796a5d7dcfe8b07b5; no voters:
I20240502 07:58:20.142901 20039 raft_consensus.cc:2781] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:20.142961 20039 raft_consensus.cc:482] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:20.142993 20039 raft_consensus.cc:3037] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:20.143903 20039 raft_consensus.cc:504] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:20.144229 20039 leader_election.cc:288] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 election: Requested vote from peers 9f21202c69df478796a5d7dcfe8b07b5 (127.18.13.131:44389), d19f943e214647a3850f51a324017662 (127.18.13.133:40945)
I20240502 07:58:20.144546 18934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fb35f752fd4e3d8e9821ee645b88e2" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21202c69df478796a5d7dcfe8b07b5"
I20240502 07:58:20.144553 19198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fb35f752fd4e3d8e9821ee645b88e2" candidate_uuid: "2f6bc76c600c413d9f72526563d3234a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19f943e214647a3850f51a324017662"
I20240502 07:58:20.144670 18934 raft_consensus.cc:3037] T 03fb35f752fd4e3d8e9821ee645b88e2 P 9f21202c69df478796a5d7dcfe8b07b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:20.144680 19198 raft_consensus.cc:3037] T 03fb35f752fd4e3d8e9821ee645b88e2 P d19f943e214647a3850f51a324017662 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:20.145435 18934 raft_consensus.cc:2454] T 03fb35f752fd4e3d8e9821ee645b88e2 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 1.
I20240502 07:58:20.145445 19198 raft_consensus.cc:2454] T 03fb35f752fd4e3d8e9821ee645b88e2 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6bc76c600c413d9f72526563d3234a in term 1.
I20240502 07:58:20.145709 18605 leader_election.cc:302] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6bc76c600c413d9f72526563d3234a, 9f21202c69df478796a5d7dcfe8b07b5; no voters:
I20240502 07:58:20.145874 20039 raft_consensus.cc:2781] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:20.145967 20039 raft_consensus.cc:686] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [term 1 LEADER]: Becoming Leader. State: Replica: 2f6bc76c600c413d9f72526563d3234a, State: Running, Role: LEADER
I20240502 07:58:20.146112 20039 consensus_queue.cc:237] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } }
I20240502 07:58:20.148058 18520 catalog_manager.cc:5484] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f6bc76c600c413d9f72526563d3234a (127.18.13.129). New cstate: current_term: 1 leader_uuid: "2f6bc76c600c413d9f72526563d3234a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6bc76c600c413d9f72526563d3234a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:20.168164 20148 raft_consensus.cc:482] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:20.168299 20148 raft_consensus.cc:504] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:20.168730 20148 leader_election.cc:288] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647), 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417)
I20240502 07:58:20.169179 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161328ecf93f48d399060275e63a8df1" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" is_pre_election: true
I20240502 07:58:20.169179 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161328ecf93f48d399060275e63a8df1" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743" is_pre_election: true
I20240502 07:58:20.169330 19331 raft_consensus.cc:2454] T 161328ecf93f48d399060275e63a8df1 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 0.
I20240502 07:58:20.169333 19066 raft_consensus.cc:2454] T 161328ecf93f48d399060275e63a8df1 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 0.
I20240502 07:58:20.169658 18738 leader_election.cc:302] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:20.169898 20148 raft_consensus.cc:2781] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:20.169958 20148 raft_consensus.cc:482] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:20.169988 20148 raft_consensus.cc:3037] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:20.170805 20148 raft_consensus.cc:504] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:20.171139 20148 leader_election.cc:288] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 election: Requested vote from peers 345ab146afdf4d39a1d8583881c06743 (127.18.13.134:44647), 341d0eb74c3544faaa2633d7aa3f77b3 (127.18.13.132:35417)
I20240502 07:58:20.171473 19331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161328ecf93f48d399060275e63a8df1" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345ab146afdf4d39a1d8583881c06743"
I20240502 07:58:20.171470 19066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161328ecf93f48d399060275e63a8df1" candidate_uuid: "4135e6d90aec4830b08bcb7e8dc50260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "341d0eb74c3544faaa2633d7aa3f77b3"
I20240502 07:58:20.171577 19331 raft_consensus.cc:3037] T 161328ecf93f48d399060275e63a8df1 P 345ab146afdf4d39a1d8583881c06743 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:20.171631 19066 raft_consensus.cc:3037] T 161328ecf93f48d399060275e63a8df1 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:20.172374 19331 raft_consensus.cc:2454] T 161328ecf93f48d399060275e63a8df1 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 1.
I20240502 07:58:20.172446 19066 raft_consensus.cc:2454] T 161328ecf93f48d399060275e63a8df1 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4135e6d90aec4830b08bcb7e8dc50260 in term 1.
I20240502 07:58:20.172776 18738 leader_election.cc:302] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 341d0eb74c3544faaa2633d7aa3f77b3, 4135e6d90aec4830b08bcb7e8dc50260; no voters:
I20240502 07:58:20.172972 20148 raft_consensus.cc:2781] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:20.173060 20148 raft_consensus.cc:686] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [term 1 LEADER]: Becoming Leader. State: Replica: 4135e6d90aec4830b08bcb7e8dc50260, State: Running, Role: LEADER
I20240502 07:58:20.173208 20148 consensus_queue.cc:237] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } }
I20240502 07:58:20.175266 18520 catalog_manager.cc:5484] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4135e6d90aec4830b08bcb7e8dc50260 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "4135e6d90aec4830b08bcb7e8dc50260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:20.313553 20207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:20.313825 20207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:20.316951 20207 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240502 07:58:20.396572 19198 raft_consensus.cc:1261] T 03fb35f752fd4e3d8e9821ee645b88e2 P d19f943e214647a3850f51a324017662 [term 1 FOLLOWER]: Refusing update from remote peer 2f6bc76c600c413d9f72526563d3234a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:20.396925 19066 raft_consensus.cc:1261] T 161328ecf93f48d399060275e63a8df1 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Refusing update from remote peer 4135e6d90aec4830b08bcb7e8dc50260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:20.397679 18934 raft_consensus.cc:1261] T 03fb35f752fd4e3d8e9821ee645b88e2 P 9f21202c69df478796a5d7dcfe8b07b5 [term 1 FOLLOWER]: Refusing update from remote peer 2f6bc76c600c413d9f72526563d3234a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:20.398015 20148 consensus_queue.cc:1035] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:20.400024 18803 raft_consensus.cc:3037] T b726b43a39a7440199728ec8d51817d8 P 4135e6d90aec4830b08bcb7e8dc50260 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:20.400959 20039 consensus_queue.cc:1035] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f21202c69df478796a5d7dcfe8b07b5" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:20.401271 20039 consensus_queue.cc:1035] T 03fb35f752fd4e3d8e9821ee645b88e2 P 2f6bc76c600c413d9f72526563d3234a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19f943e214647a3850f51a324017662" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:20.401274 19331 raft_consensus.cc:1261] T 161328ecf93f48d399060275e63a8df1 P 345ab146afdf4d39a1d8583881c06743 [term 1 FOLLOWER]: Refusing update from remote peer 4135e6d90aec4830b08bcb7e8dc50260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:20.401743 20148 consensus_queue.cc:1035] T 161328ecf93f48d399060275e63a8df1 P 4135e6d90aec4830b08bcb7e8dc50260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "345ab146afdf4d39a1d8583881c06743" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:20.408901 19067 raft_consensus.cc:1261] T b726b43a39a7440199728ec8d51817d8 P 341d0eb74c3544faaa2633d7aa3f77b3 [term 1 FOLLOWER]: Refusing update from remote peer 345ab146afdf4d39a1d8583881c06743: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:20.409335 20100 consensus_queue.cc:1035] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341d0eb74c3544faaa2633d7aa3f77b3" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:20.412791 20100 consensus_queue.cc:1035] T b726b43a39a7440199728ec8d51817d8 P 345ab146afdf4d39a1d8583881c06743 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4135e6d90aec4830b08bcb7e8dc50260" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 114.824 ms
time per row: 0.0574119 ms
I20240502 07:58:20.697256 18783 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:20.697660 19311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:58:20.698055 18651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:20.698649 19046 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:20.700965 18915 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 07:58:20.725383 19179 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:20.841596 18783 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:20.843636 18651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:20.846634 18915 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 07:58:20.846904 19311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:58:20.856894 19046 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:20.867558 19179 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:20.999186 18651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:20.999994 18783 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:21.000353 19311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:58:21.001089 19046 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:21.001648 18915 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 07:58:21.014735 19179 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:21.131855 18651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:21.132406 19046 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:21.133105 18783 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:21.134682 19311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:58:21.143868 18915 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 07:58:21.158633 19179 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:21.266564 19046 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:21.266891 18651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:21.273305 19179 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:21.273871 19311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:58:21.284094 18915 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 07:58:21.287292 18783 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:21.428138 19311 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:58:21.428787 18915 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 07:58:21.429561 18651 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:21.443420 19179 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:21.444041 19046 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:21.445612 18783 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
7375e73262114a86b5bdd384b5c786ee | 127.18.13.190:46263 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.13.148:34465 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2f6bc76c600c413d9f72526563d3234a | 127.18.13.129:40523 | HEALTHY | /A | 6 | 0
341d0eb74c3544faaa2633d7aa3f77b3 | 127.18.13.132:35417 | HEALTHY | /B | 6 | 0
345ab146afdf4d39a1d8583881c06743 | 127.18.13.134:44647 | HEALTHY | /C | 7 | 0
4135e6d90aec4830b08bcb7e8dc50260 | 127.18.13.130:43999 | HEALTHY | /A | 6 | 0
9f21202c69df478796a5d7dcfe8b07b5 | 127.18.13.131:44389 | HEALTHY | /B | 8 | 0
d19f943e214647a3850f51a324017662 | 127.18.13.133:40945 | HEALTHY | /C | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/C | 2
/A | 2
/B | 2
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.18.13.129 | experimental | 127.18.13.129:40523
local_ip_for_outbound_sockets | 127.18.13.130 | experimental | 127.18.13.130:43999
local_ip_for_outbound_sockets | 127.18.13.131 | experimental | 127.18.13.131:44389
local_ip_for_outbound_sockets | 127.18.13.132 | experimental | 127.18.13.132:35417
local_ip_for_outbound_sockets | 127.18.13.133 | experimental | 127.18.13.133:40945
local_ip_for_outbound_sockets | 127.18.13.134 | experimental | 127.18.13.134:44647
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.13.129:40523
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.13.130:43999
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.13.131:44389
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.18.13.132:35417
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.13.133:40945
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.13.134:44647
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.13.148:34465 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_10 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_11 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 18
Median | 18
Third Quartile | 18
Maximum | 18
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 12
Tablets | 36
Replicas | 108
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 07:58:21.489559 18486 log_verifier.cc:126] Checking tablet 03fb35f752fd4e3d8e9821ee645b88e2
I20240502 07:58:21.517503 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 03fb35f752fd4e3d8e9821ee645b88e2
I20240502 07:58:21.517594 18486 log_verifier.cc:126] Checking tablet 062558123ed44b04969b84580b8678c9
I20240502 07:58:21.542074 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 062558123ed44b04969b84580b8678c9
I20240502 07:58:21.542143 18486 log_verifier.cc:126] Checking tablet 07fd48689c0940909626a09af99e954a
I20240502 07:58:21.566232 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 07fd48689c0940909626a09af99e954a
I20240502 07:58:21.566303 18486 log_verifier.cc:126] Checking tablet 08f66fc67af845b08eaf00717e75d667
I20240502 07:58:21.590655 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 08f66fc67af845b08eaf00717e75d667
I20240502 07:58:21.590728 18486 log_verifier.cc:126] Checking tablet 112385d70673444fb1c253255d47d8f8
I20240502 07:58:21.615137 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 112385d70673444fb1c253255d47d8f8
I20240502 07:58:21.615200 18486 log_verifier.cc:126] Checking tablet 161328ecf93f48d399060275e63a8df1
I20240502 07:58:21.639721 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 161328ecf93f48d399060275e63a8df1
I20240502 07:58:21.639808 18486 log_verifier.cc:126] Checking tablet 275d344bb7e044db9f38f9d1215a0c2a
I20240502 07:58:21.664620 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 275d344bb7e044db9f38f9d1215a0c2a
I20240502 07:58:21.664696 18486 log_verifier.cc:126] Checking tablet 431f2dd73e8c42fcb3b5ba171278fbc3
I20240502 07:58:21.689077 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 431f2dd73e8c42fcb3b5ba171278fbc3
I20240502 07:58:21.689157 18486 log_verifier.cc:126] Checking tablet 43a36fb737fe418aa662fc75bcf5a3da
I20240502 07:58:21.713712 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 43a36fb737fe418aa662fc75bcf5a3da
I20240502 07:58:21.713814 18486 log_verifier.cc:126] Checking tablet 46247e07dbff4195b8681adb87bccbce
I20240502 07:58:21.737834 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 46247e07dbff4195b8681adb87bccbce
I20240502 07:58:21.737901 18486 log_verifier.cc:126] Checking tablet 518afbe60085421aa353b7a7ac13d4d9
I20240502 07:58:21.762815 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 518afbe60085421aa353b7a7ac13d4d9
I20240502 07:58:21.762899 18486 log_verifier.cc:126] Checking tablet 5272d85c6ec1410a9b5e7ab3c0b9bfb2
I20240502 07:58:21.787082 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5272d85c6ec1410a9b5e7ab3c0b9bfb2
I20240502 07:58:21.787168 18486 log_verifier.cc:126] Checking tablet 58e1075a0a4f481c81181f7120dcb798
I20240502 07:58:21.810734 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 58e1075a0a4f481c81181f7120dcb798
I20240502 07:58:21.810806 18486 log_verifier.cc:126] Checking tablet 6a567790cbaf4fc585a0c3276b7ac6d7
I20240502 07:58:21.837026 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6a567790cbaf4fc585a0c3276b7ac6d7
I20240502 07:58:21.837113 18486 log_verifier.cc:126] Checking tablet 6f1cbd7ae66a4d25a82a5020b1d70fb6
I20240502 07:58:21.865326 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6f1cbd7ae66a4d25a82a5020b1d70fb6
I20240502 07:58:21.865397 18486 log_verifier.cc:126] Checking tablet 7ce2abb8fd5345b1ba544ba04b82e2ea
I20240502 07:58:21.890544 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7ce2abb8fd5345b1ba544ba04b82e2ea
I20240502 07:58:21.890628 18486 log_verifier.cc:126] Checking tablet 7d7eaa100f764d898c106360d07363fa
I20240502 07:58:21.915179 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7d7eaa100f764d898c106360d07363fa
I20240502 07:58:21.915244 18486 log_verifier.cc:126] Checking tablet 7fc479ebe934455aa25a07880c812a24
I20240502 07:58:21.940057 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7fc479ebe934455aa25a07880c812a24
I20240502 07:58:21.940126 18486 log_verifier.cc:126] Checking tablet 822ecfcf16fa4c60897cc18d14138289
I20240502 07:58:21.964821 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 822ecfcf16fa4c60897cc18d14138289
I20240502 07:58:21.964890 18486 log_verifier.cc:126] Checking tablet 84aed0b3a5e74500b8646d0dbe9b26a1
I20240502 07:58:21.989465 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 84aed0b3a5e74500b8646d0dbe9b26a1
I20240502 07:58:21.989532 18486 log_verifier.cc:126] Checking tablet 887023183cdc419c9f3d4f6b7bd21fe0
I20240502 07:58:22.014086 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 887023183cdc419c9f3d4f6b7bd21fe0
I20240502 07:58:22.014153 18486 log_verifier.cc:126] Checking tablet 8bc3357e14f24f0682bebbbba109d401
I20240502 07:58:22.039528 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8bc3357e14f24f0682bebbbba109d401
I20240502 07:58:22.039608 18486 log_verifier.cc:126] Checking tablet 9656cbeef84441f3ba7be5301d1ba339
I20240502 07:58:22.064199 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9656cbeef84441f3ba7be5301d1ba339
I20240502 07:58:22.064273 18486 log_verifier.cc:126] Checking tablet 96cddef766ee4366991da031cbc8e2d5
I20240502 07:58:22.088433 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 96cddef766ee4366991da031cbc8e2d5
I20240502 07:58:22.088506 18486 log_verifier.cc:126] Checking tablet 97761e6b14b047f1a598750d405063db
I20240502 07:58:22.112864 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 97761e6b14b047f1a598750d405063db
I20240502 07:58:22.112931 18486 log_verifier.cc:126] Checking tablet a0ae282afe034174bb547fe021b725ae
I20240502 07:58:22.138258 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a0ae282afe034174bb547fe021b725ae
I20240502 07:58:22.138332 18486 log_verifier.cc:126] Checking tablet b726b43a39a7440199728ec8d51817d8
I20240502 07:58:22.162752 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b726b43a39a7440199728ec8d51817d8
I20240502 07:58:22.162837 18486 log_verifier.cc:126] Checking tablet b84243b4d86d49afa04c6abbc2a36bd1
I20240502 07:58:22.186949 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b84243b4d86d49afa04c6abbc2a36bd1
I20240502 07:58:22.187017 18486 log_verifier.cc:126] Checking tablet c7376645e76b4c5abdcd721bea1dae5d
I20240502 07:58:22.210995 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c7376645e76b4c5abdcd721bea1dae5d
I20240502 07:58:22.211058 18486 log_verifier.cc:126] Checking tablet cb668216cb3641f6a108a60b74717a70
I20240502 07:58:22.235219 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet cb668216cb3641f6a108a60b74717a70
I20240502 07:58:22.235296 18486 log_verifier.cc:126] Checking tablet e128121cf15a4b5aa3f07265ed60371f
I20240502 07:58:22.260169 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e128121cf15a4b5aa3f07265ed60371f
I20240502 07:58:22.260236 18486 log_verifier.cc:126] Checking tablet f0b7aba425554d5f98024438383bbf37
I20240502 07:58:22.284327 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f0b7aba425554d5f98024438383bbf37
I20240502 07:58:22.284401 18486 log_verifier.cc:126] Checking tablet fc354000a8404e7c95a2ca5f5565c2ce
I20240502 07:58:22.308674 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fc354000a8404e7c95a2ca5f5565c2ce
I20240502 07:58:22.308743 18486 log_verifier.cc:126] Checking tablet fce88a610b8747d090d7fdda3d782abb
I20240502 07:58:22.333436 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fce88a610b8747d090d7fdda3d782abb
I20240502 07:58:22.333510 18486 log_verifier.cc:126] Checking tablet fe229d37d9f44877a5b2720f3a7ef6cd
I20240502 07:58:22.358031 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fe229d37d9f44877a5b2720f3a7ef6cd
I20240502 07:58:22.358098 18486 log_verifier.cc:126] Checking tablet ff4246e864f649fdb53bc67cae2a49a7
I20240502 07:58:22.381726 18486 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ff4246e864f649fdb53bc67cae2a49a7
I20240502 07:58:22.394588 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 18589
I20240502 07:58:22.401749 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 18720
I20240502 07:58:22.409276 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 18852
I20240502 07:58:22.416479 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 18984
W20240502 07:58:22.418359 19266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.13.129:40523: connect: Connection refused (error 111)
W20240502 07:58:22.419234 19266 consensus_peers.cc:485] T 6f1cbd7ae66a4d25a82a5020b1d70fb6 P 345ab146afdf4d39a1d8583881c06743 -> Peer 2f6bc76c600c413d9f72526563d3234a (127.18.13.129:40523): Couldn't send request to peer 2f6bc76c600c413d9f72526563d3234a. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.129:40523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:22.423009 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 19117
I20240502 07:58:22.429282 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 19248
I20240502 07:58:22.435365 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 18498
2024-05-02T07:58:22Z chronyd exiting
[ OK ] LocationAwareRebalancingBasicTest.Basic (13925 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (13925 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240502 07:58:22.555558 18486 test_util.cc:268] Using random seed: -141353567
I20240502 07:58:22.556388 18486 ts_itest-base.cc:112] Starting cluster with:
I20240502 07:58:22.556428 18486 ts_itest-base.cc:113] --------------
I20240502 07:58:22.556449 18486 ts_itest-base.cc:114] 5 tablet servers
I20240502 07:58:22.556466 18486 ts_itest-base.cc:115] 3 replicas per TS
I20240502 07:58:22.556491 18486 ts_itest-base.cc:116] --------------
2024-05-02T07:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T07:58:22Z Disabled control of system clock
I20240502 07:58:22.564915 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.13.190:46473
--webserver_interface=127.18.13.190
--webserver_port=0
--builtin_ntp_servers=127.18.13.148:36487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.13.190:46473
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 07:58:22.673859 20543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:22.674098 20543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:22.674137 20543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:22.677188 20543 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:22.677263 20543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 07:58:22.677284 20543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:22.677304 20543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 07:58:22.677340 20543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 07:58:22.681095 20543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:36487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.18.13.190:46473
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.13.190:46473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.18.13.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.20543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:22.682042 20543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:22.683168 20543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:22.691468 20548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:22.692025 20549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:22.693269 20543 server_base.cc:1034] running on GCE node
W20240502 07:58:22.694289 20551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:22.694871 20543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:22.695967 20543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:22.697115 20543 hybrid_clock.cc:648] HybridClock initialized: now 1714636702697107 us; error 34 us; skew 500 ppm
I20240502 07:58:22.699169 20543 webserver.cc:460] Webserver started at http://127.18.13.190:46581/ using document root <none> and password file <none>
I20240502 07:58:22.699748 20543 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:22.699838 20543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:22.700032 20543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:22.701758 20543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "59fb737ddd43406c80289a051971b055"
format_stamp: "Formatted at 2024-05-02 07:58:22 on dist-test-slave-f46z"
I20240502 07:58:22.702226 20543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "59fb737ddd43406c80289a051971b055"
format_stamp: "Formatted at 2024-05-02 07:58:22 on dist-test-slave-f46z"
I20240502 07:58:22.705678 20543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20240502 07:58:22.708060 20557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:22.709179 20543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240502 07:58:22.709308 20543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
uuid: "59fb737ddd43406c80289a051971b055"
format_stamp: "Formatted at 2024-05-02 07:58:22 on dist-test-slave-f46z"
I20240502 07:58:22.709399 20543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:22.723757 20543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:22.724572 20543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:22.724717 20543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:22.731518 20609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.190:46473 every 8 connection(s)
I20240502 07:58:22.731520 20543 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.190:46473
I20240502 07:58:22.732609 20543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 07:58:22.735244 20610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:22.740569 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 20543
I20240502 07:58:22.740629 20610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055: Bootstrap starting.
I20240502 07:58:22.740694 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 07:58:22.743424 20610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:22.744439 20610 log.cc:824] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:22.746363 20610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055: No bootstrap required, opened a new log
I20240502 07:58:22.749552 20610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fb737ddd43406c80289a051971b055" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46473 } }
I20240502 07:58:22.749775 20610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:22.749846 20610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59fb737ddd43406c80289a051971b055, State: Initialized, Role: FOLLOWER
I20240502 07:58:22.750398 20610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fb737ddd43406c80289a051971b055" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46473 } }
I20240502 07:58:22.750547 20610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:22.750622 20610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:22.750751 20610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:22.751744 20610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fb737ddd43406c80289a051971b055" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46473 } }
I20240502 07:58:22.752135 20610 leader_election.cc:302] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59fb737ddd43406c80289a051971b055; no voters:
I20240502 07:58:22.752454 20610 leader_election.cc:288] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:58:22.752595 20615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:22.753136 20615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [term 1 LEADER]: Becoming Leader. State: Replica: 59fb737ddd43406c80289a051971b055, State: Running, Role: LEADER
I20240502 07:58:22.753451 20615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fb737ddd43406c80289a051971b055" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46473 } }
I20240502 07:58:22.753656 20610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:58:22.754987 20616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59fb737ddd43406c80289a051971b055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fb737ddd43406c80289a051971b055" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46473 } } }
I20240502 07:58:22.755111 20616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:22.754971 20617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59fb737ddd43406c80289a051971b055. Latest consensus state: current_term: 1 leader_uuid: "59fb737ddd43406c80289a051971b055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59fb737ddd43406c80289a051971b055" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46473 } } }
I20240502 07:58:22.755396 20617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055 [sys.catalog]: This master's current role is: LEADER
W20240502 07:58:22.758267 20630 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 07:58:22.758348 20630 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 07:58:22.758441 20626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:58:22.759354 20626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:58:22.764271 20626 catalog_manager.cc:1347] Generated new cluster ID: 6405ec0a40b745e3b4171d04820fc269
I20240502 07:58:22.764348 20626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:58:22.781128 20626 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:58:22.782076 20626 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:58:22.786710 20626 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 59fb737ddd43406c80289a051971b055: Generated new TSK 0
I20240502 07:58:22.787395 20626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:58:22.794839 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.129:0
--local_ip_for_outbound_sockets=127.18.13.129
--webserver_interface=127.18.13.129
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46473
--builtin_ntp_servers=127.18.13.148:36487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 07:58:22.905478 20634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:22.905740 20634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:22.905825 20634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:22.908941 20634 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:22.909041 20634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:22.909152 20634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.129
I20240502 07:58:22.912964 20634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:36487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.18.13.129
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46473
--never_fsync=true
--heap_profile_path=/tmp/kudu.20634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.129
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:22.913967 20634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:22.915155 20634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:22.924110 20642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:22.924599 20639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:22.925884 20640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:22.926970 20634 server_base.cc:1034] running on GCE node
I20240502 07:58:22.927428 20634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:22.928081 20634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:22.929246 20634 hybrid_clock.cc:648] HybridClock initialized: now 1714636702929232 us; error 27 us; skew 500 ppm
I20240502 07:58:22.931190 20634 webserver.cc:460] Webserver started at http://127.18.13.129:44569/ using document root <none> and password file <none>
I20240502 07:58:22.931746 20634 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:22.931839 20634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:22.932046 20634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:22.933712 20634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6c7c67e750b543238fa50260c28cca75"
format_stamp: "Formatted at 2024-05-02 07:58:22 on dist-test-slave-f46z"
I20240502 07:58:22.934197 20634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6c7c67e750b543238fa50260c28cca75"
format_stamp: "Formatted at 2024-05-02 07:58:22 on dist-test-slave-f46z"
I20240502 07:58:22.938041 20634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240502 07:58:22.940508 20648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:22.941628 20634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 07:58:22.941742 20634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6c7c67e750b543238fa50260c28cca75"
format_stamp: "Formatted at 2024-05-02 07:58:22 on dist-test-slave-f46z"
I20240502 07:58:22.941833 20634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:22.964056 20634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:22.964676 20634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:22.964845 20634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:22.965389 20634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:22.966306 20634 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:22.966357 20634 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:22.966418 20634 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:22.966447 20634 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:22.974853 20634 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.129:45525
I20240502 07:58:22.974970 20761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.129:45525 every 8 connection(s)
I20240502 07:58:22.975870 20634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 07:58:22.980995 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 20634
I20240502 07:58:22.981138 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 07:58:22.983470 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.130:0
--local_ip_for_outbound_sockets=127.18.13.130
--webserver_interface=127.18.13.130
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46473
--builtin_ntp_servers=127.18.13.148:36487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:22.985407 20762 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46473
I20240502 07:58:22.985663 20762 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:22.986346 20762 heartbeater.cc:511] Master 127.18.13.190:46473 requested a full tablet report, sending...
I20240502 07:58:22.988029 20574 ts_manager.cc:194] Registered new tserver with Master: 6c7c67e750b543238fa50260c28cca75 (127.18.13.129:45525)
I20240502 07:58:22.989214 20574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.129:53619
W20240502 07:58:23.096800 20765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:23.097080 20765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:23.097162 20765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:23.100287 20765 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:23.100374 20765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:23.100492 20765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.130
I20240502 07:58:23.104211 20765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:36487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.18.13.130
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46473
--never_fsync=true
--heap_profile_path=/tmp/kudu.20765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.130
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:23.105188 20765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:23.106354 20765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:23.115559 20770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:23.116313 20773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:23.116956 20771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:23.118096 20765 server_base.cc:1034] running on GCE node
I20240502 07:58:23.118489 20765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:23.119067 20765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:23.120236 20765 hybrid_clock.cc:648] HybridClock initialized: now 1714636703120210 us; error 43 us; skew 500 ppm
I20240502 07:58:23.122162 20765 webserver.cc:460] Webserver started at http://127.18.13.130:36737/ using document root <none> and password file <none>
I20240502 07:58:23.122699 20765 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:23.122761 20765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:23.122938 20765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:23.124622 20765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f4844e363fbb401e8185a120d15cb281"
format_stamp: "Formatted at 2024-05-02 07:58:23 on dist-test-slave-f46z"
I20240502 07:58:23.125115 20765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f4844e363fbb401e8185a120d15cb281"
format_stamp: "Formatted at 2024-05-02 07:58:23 on dist-test-slave-f46z"
I20240502 07:58:23.128424 20765 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 07:58:23.130734 20779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:23.131713 20765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:23.131839 20765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f4844e363fbb401e8185a120d15cb281"
format_stamp: "Formatted at 2024-05-02 07:58:23 on dist-test-slave-f46z"
I20240502 07:58:23.131925 20765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:23.146117 20765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:23.146776 20765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:23.146942 20765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:23.147490 20765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:23.148416 20765 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:23.148468 20765 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:23.148527 20765 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:23.148556 20765 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:23.157035 20765 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.130:40761
I20240502 07:58:23.157091 20892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.130:40761 every 8 connection(s)
I20240502 07:58:23.157951 20765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 07:58:23.159943 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 20765
I20240502 07:58:23.160037 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 07:58:23.162309 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.131:0
--local_ip_for_outbound_sockets=127.18.13.131
--webserver_interface=127.18.13.131
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46473
--builtin_ntp_servers=127.18.13.148:36487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:23.167369 20893 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46473
I20240502 07:58:23.167637 20893 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:23.168371 20893 heartbeater.cc:511] Master 127.18.13.190:46473 requested a full tablet report, sending...
I20240502 07:58:23.169528 20574 ts_manager.cc:194] Registered new tserver with Master: f4844e363fbb401e8185a120d15cb281 (127.18.13.130:40761)
I20240502 07:58:23.170073 20574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.130:50179
W20240502 07:58:23.271903 20896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:23.272138 20896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:23.272197 20896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:23.275131 20896 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:23.275189 20896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:23.275270 20896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.131
I20240502 07:58:23.278795 20896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:36487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.18.13.131
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46473
--never_fsync=true
--heap_profile_path=/tmp/kudu.20896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.131
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:23.279644 20896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:23.280757 20896 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:23.289554 20901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:23.290146 20902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:23.291282 20904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:23.292500 20896 server_base.cc:1034] running on GCE node
I20240502 07:58:23.292965 20896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:23.293555 20896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:23.294713 20896 hybrid_clock.cc:648] HybridClock initialized: now 1714636703294691 us; error 39 us; skew 500 ppm
I20240502 07:58:23.296751 20896 webserver.cc:460] Webserver started at http://127.18.13.131:43199/ using document root <none> and password file <none>
I20240502 07:58:23.297325 20896 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:23.297389 20896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:23.297593 20896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:23.299269 20896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fc87c2ec1ca742af906554fe00e7bbde"
format_stamp: "Formatted at 2024-05-02 07:58:23 on dist-test-slave-f46z"
I20240502 07:58:23.299723 20896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fc87c2ec1ca742af906554fe00e7bbde"
format_stamp: "Formatted at 2024-05-02 07:58:23 on dist-test-slave-f46z"
I20240502 07:58:23.303020 20896 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:23.305217 20910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:23.306325 20896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:23.306432 20896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fc87c2ec1ca742af906554fe00e7bbde"
format_stamp: "Formatted at 2024-05-02 07:58:23 on dist-test-slave-f46z"
I20240502 07:58:23.306519 20896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:23.320797 20896 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:23.321413 20896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:23.321570 20896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:23.322098 20896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:23.322973 20896 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:23.323021 20896 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:23.323081 20896 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:23.323110 20896 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:23.331375 20896 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.131:36107
I20240502 07:58:23.331395 21023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.131:36107 every 8 connection(s)
I20240502 07:58:23.332363 20896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 07:58:23.338335 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 20896
I20240502 07:58:23.338456 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 07:58:23.340631 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.132:0
--local_ip_for_outbound_sockets=127.18.13.132
--webserver_interface=127.18.13.132
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46473
--builtin_ntp_servers=127.18.13.148:36487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:23.340991 21024 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46473
I20240502 07:58:23.341187 21024 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:23.341667 21024 heartbeater.cc:511] Master 127.18.13.190:46473 requested a full tablet report, sending...
I20240502 07:58:23.342586 20574 ts_manager.cc:194] Registered new tserver with Master: fc87c2ec1ca742af906554fe00e7bbde (127.18.13.131:36107)
I20240502 07:58:23.343115 20574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.131:33157
W20240502 07:58:23.446097 21027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:23.446375 21027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:23.446462 21027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:23.449472 21027 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:23.449546 21027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:23.449651 21027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.132
I20240502 07:58:23.453222 21027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:36487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.18.13.132
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46473
--never_fsync=true
--heap_profile_path=/tmp/kudu.21027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.132
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:23.454203 21027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:23.455402 21027 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:23.464068 21032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:23.464591 21033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:23.465970 21035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:23.466961 21027 server_base.cc:1034] running on GCE node
I20240502 07:58:23.467356 21027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:23.467947 21027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:23.469095 21027 hybrid_clock.cc:648] HybridClock initialized: now 1714636703469088 us; error 31 us; skew 500 ppm
I20240502 07:58:23.471007 21027 webserver.cc:460] Webserver started at http://127.18.13.132:37673/ using document root <none> and password file <none>
I20240502 07:58:23.471561 21027 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:23.471625 21027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:23.471825 21027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:23.473474 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4a323a9af840484b803348390bbf026d"
format_stamp: "Formatted at 2024-05-02 07:58:23 on dist-test-slave-f46z"
I20240502 07:58:23.473953 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4a323a9af840484b803348390bbf026d"
format_stamp: "Formatted at 2024-05-02 07:58:23 on dist-test-slave-f46z"
I20240502 07:58:23.477232 21027 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240502 07:58:23.479341 21041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:23.480329 21027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:23.480432 21027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4a323a9af840484b803348390bbf026d"
format_stamp: "Formatted at 2024-05-02 07:58:23 on dist-test-slave-f46z"
I20240502 07:58:23.480516 21027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:23.517387 21027 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:23.518075 21027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:23.518247 21027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:23.518796 21027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:23.519704 21027 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:23.519757 21027 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:23.519834 21027 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:23.519868 21027 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:23.528146 21027 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.132:45233
I20240502 07:58:23.528172 21154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.132:45233 every 8 connection(s)
I20240502 07:58:23.529068 21027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:58:23.536968 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 21027
I20240502 07:58:23.537089 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 07:58:23.537900 21155 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46473
I20240502 07:58:23.538130 21155 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:23.538708 21155 heartbeater.cc:511] Master 127.18.13.190:46473 requested a full tablet report, sending...
I20240502 07:58:23.539177 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.133:0
--local_ip_for_outbound_sockets=127.18.13.133
--webserver_interface=127.18.13.133
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46473
--builtin_ntp_servers=127.18.13.148:36487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:23.539772 20574 ts_manager.cc:194] Registered new tserver with Master: 4a323a9af840484b803348390bbf026d (127.18.13.132:45233)
I20240502 07:58:23.540549 20574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.132:60665
W20240502 07:58:23.650018 21158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:23.650266 21158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:23.650313 21158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:23.653365 21158 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:23.653435 21158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:23.653506 21158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.133
I20240502 07:58:23.657291 21158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:36487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.18.13.133
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46473
--never_fsync=true
--heap_profile_path=/tmp/kudu.21158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.133
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:23.658165 21158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:23.659328 21158 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:23.669266 21163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:23.669688 21164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:23.670976 21166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:23.672053 21158 server_base.cc:1034] running on GCE node
I20240502 07:58:23.672489 21158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:23.673085 21158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:23.674253 21158 hybrid_clock.cc:648] HybridClock initialized: now 1714636703674234 us; error 34 us; skew 500 ppm
I20240502 07:58:23.676324 21158 webserver.cc:460] Webserver started at http://127.18.13.133:39147/ using document root <none> and password file <none>
I20240502 07:58:23.676903 21158 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:23.676972 21158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:23.677147 21158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:23.678803 21158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a239c2e12edf43c383283f847a7a5c8e"
format_stamp: "Formatted at 2024-05-02 07:58:23 on dist-test-slave-f46z"
I20240502 07:58:23.679284 21158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a239c2e12edf43c383283f847a7a5c8e"
format_stamp: "Formatted at 2024-05-02 07:58:23 on dist-test-slave-f46z"
I20240502 07:58:23.682652 21158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 07:58:23.685079 21172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:23.686121 21158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:23.686239 21158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a239c2e12edf43c383283f847a7a5c8e"
format_stamp: "Formatted at 2024-05-02 07:58:23 on dist-test-slave-f46z"
I20240502 07:58:23.686337 21158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:23.708758 21158 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:23.709430 21158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:23.709618 21158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:23.710214 21158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:23.711158 21158 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:23.711213 21158 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:23.711274 21158 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:23.711314 21158 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:23.719780 21158 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.133:38353
I20240502 07:58:23.719904 21285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.133:38353 every 8 connection(s)
I20240502 07:58:23.720976 21158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:58:23.725865 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 21158
I20240502 07:58:23.726014 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 07:58:23.730451 21286 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46473
I20240502 07:58:23.730697 21286 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:23.731403 21286 heartbeater.cc:511] Master 127.18.13.190:46473 requested a full tablet report, sending...
I20240502 07:58:23.732429 20574 ts_manager.cc:194] Registered new tserver with Master: a239c2e12edf43c383283f847a7a5c8e (127.18.13.133:38353)
I20240502 07:58:23.732978 20574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.133:48383
I20240502 07:58:23.740620 18486 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 07:58:23.751760 20574 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37020:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 07:58:23.770604 21220 tablet_service.cc:1467] Processing CreateTablet for tablet 4f4432df829b4d21a2d6c8840ca8ebd1 (DEFAULT_TABLE table=TestTable [id=d4db6e7f7bc044bcb923ee7e01278cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:23.770604 20958 tablet_service.cc:1467] Processing CreateTablet for tablet 4f4432df829b4d21a2d6c8840ca8ebd1 (DEFAULT_TABLE table=TestTable [id=d4db6e7f7bc044bcb923ee7e01278cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:23.770604 20696 tablet_service.cc:1467] Processing CreateTablet for tablet 4f4432df829b4d21a2d6c8840ca8ebd1 (DEFAULT_TABLE table=TestTable [id=d4db6e7f7bc044bcb923ee7e01278cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:23.771634 21220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4432df829b4d21a2d6c8840ca8ebd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:23.771636 20696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4432df829b4d21a2d6c8840ca8ebd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:23.771636 20958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f4432df829b4d21a2d6c8840ca8ebd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:23.777171 21303 tablet_bootstrap.cc:492] T 4f4432df829b4d21a2d6c8840ca8ebd1 P a239c2e12edf43c383283f847a7a5c8e: Bootstrap starting.
I20240502 07:58:23.778267 21305 tablet_bootstrap.cc:492] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde: Bootstrap starting.
I20240502 07:58:23.779371 21303 tablet_bootstrap.cc:654] T 4f4432df829b4d21a2d6c8840ca8ebd1 P a239c2e12edf43c383283f847a7a5c8e: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:23.780015 21305 tablet_bootstrap.cc:654] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:23.780244 21303 log.cc:824] T 4f4432df829b4d21a2d6c8840ca8ebd1 P a239c2e12edf43c383283f847a7a5c8e: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:23.780725 21305 log.cc:824] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:23.780895 21304 tablet_bootstrap.cc:492] T 4f4432df829b4d21a2d6c8840ca8ebd1 P 6c7c67e750b543238fa50260c28cca75: Bootstrap starting.
I20240502 07:58:23.782102 21303 tablet_bootstrap.cc:492] T 4f4432df829b4d21a2d6c8840ca8ebd1 P a239c2e12edf43c383283f847a7a5c8e: No bootstrap required, opened a new log
I20240502 07:58:23.782135 21305 tablet_bootstrap.cc:492] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde: No bootstrap required, opened a new log
I20240502 07:58:23.782290 21305 ts_tablet_manager.cc:1367] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 07:58:23.782289 21303 ts_tablet_manager.cc:1367] T 4f4432df829b4d21a2d6c8840ca8ebd1 P a239c2e12edf43c383283f847a7a5c8e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 07:58:23.782994 21304 tablet_bootstrap.cc:654] T 4f4432df829b4d21a2d6c8840ca8ebd1 P 6c7c67e750b543238fa50260c28cca75: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:23.783861 21304 log.cc:824] T 4f4432df829b4d21a2d6c8840ca8ebd1 P 6c7c67e750b543238fa50260c28cca75: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:23.785751 21304 tablet_bootstrap.cc:492] T 4f4432df829b4d21a2d6c8840ca8ebd1 P 6c7c67e750b543238fa50260c28cca75: No bootstrap required, opened a new log
I20240502 07:58:23.785699 21305 raft_consensus.cc:348] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a239c2e12edf43c383283f847a7a5c8e" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38353 } } peers { permanent_uuid: "fc87c2ec1ca742af906554fe00e7bbde" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 36107 } } peers { permanent_uuid: "6c7c67e750b543238fa50260c28cca75" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 45525 } }
I20240502 07:58:23.785970 21305 raft_consensus.cc:374] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:23.786072 21305 raft_consensus.cc:729] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc87c2ec1ca742af906554fe00e7bbde, State: Initialized, Role: FOLLOWER
I20240502 07:58:23.786093 21304 ts_tablet_manager.cc:1367] T 4f4432df829b4d21a2d6c8840ca8ebd1 P 6c7c67e750b543238fa50260c28cca75: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 07:58:23.786064 21303 raft_consensus.cc:348] T 4f4432df829b4d21a2d6c8840ca8ebd1 P a239c2e12edf43c383283f847a7a5c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a239c2e12edf43c383283f847a7a5c8e" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38353 } } peers { permanent_uuid: "fc87c2ec1ca742af906554fe00e7bbde" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 36107 } } peers { permanent_uuid: "6c7c67e750b543238fa50260c28cca75" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 45525 } }
I20240502 07:58:23.786278 21303 raft_consensus.cc:374] T 4f4432df829b4d21a2d6c8840ca8ebd1 P a239c2e12edf43c383283f847a7a5c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:23.786336 21303 raft_consensus.cc:729] T 4f4432df829b4d21a2d6c8840ca8ebd1 P a239c2e12edf43c383283f847a7a5c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a239c2e12edf43c383283f847a7a5c8e, State: Initialized, Role: FOLLOWER
I20240502 07:58:23.786598 21305 consensus_queue.cc:260] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a239c2e12edf43c383283f847a7a5c8e" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38353 } } peers { permanent_uuid: "fc87c2ec1ca742af906554fe00e7bbde" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 36107 } } peers { permanent_uuid: "6c7c67e750b543238fa50260c28cca75" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 45525 } }
I20240502 07:58:23.786732 21303 consensus_queue.cc:260] T 4f4432df829b4d21a2d6c8840ca8ebd1 P a239c2e12edf43c383283f847a7a5c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a239c2e12edf43c383283f847a7a5c8e" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38353 } } peers { permanent_uuid: "fc87c2ec1ca742af906554fe00e7bbde" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 36107 } } peers { permanent_uuid: "6c7c67e750b543238fa50260c28cca75" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 45525 } }
I20240502 07:58:23.787416 21286 heartbeater.cc:503] Master 127.18.13.190:46473 was elected leader, sending a full tablet report...
I20240502 07:58:23.787454 21305 ts_tablet_manager.cc:1398] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 07:58:23.787561 21303 ts_tablet_manager.cc:1398] T 4f4432df829b4d21a2d6c8840ca8ebd1 P a239c2e12edf43c383283f847a7a5c8e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 07:58:23.787757 21024 heartbeater.cc:503] Master 127.18.13.190:46473 was elected leader, sending a full tablet report...
I20240502 07:58:23.789505 21304 raft_consensus.cc:348] T 4f4432df829b4d21a2d6c8840ca8ebd1 P 6c7c67e750b543238fa50260c28cca75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a239c2e12edf43c383283f847a7a5c8e" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38353 } } peers { permanent_uuid: "fc87c2ec1ca742af906554fe00e7bbde" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 36107 } } peers { permanent_uuid: "6c7c67e750b543238fa50260c28cca75" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 45525 } }
I20240502 07:58:23.789708 21304 raft_consensus.cc:374] T 4f4432df829b4d21a2d6c8840ca8ebd1 P 6c7c67e750b543238fa50260c28cca75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:23.789769 21304 raft_consensus.cc:729] T 4f4432df829b4d21a2d6c8840ca8ebd1 P 6c7c67e750b543238fa50260c28cca75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7c67e750b543238fa50260c28cca75, State: Initialized, Role: FOLLOWER
I20240502 07:58:23.790163 21304 consensus_queue.cc:260] T 4f4432df829b4d21a2d6c8840ca8ebd1 P 6c7c67e750b543238fa50260c28cca75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a239c2e12edf43c383283f847a7a5c8e" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38353 } } peers { permanent_uuid: "fc87c2ec1ca742af906554fe00e7bbde" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 36107 } } peers { permanent_uuid: "6c7c67e750b543238fa50260c28cca75" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 45525 } }
I20240502 07:58:23.791002 21304 ts_tablet_manager.cc:1398] T 4f4432df829b4d21a2d6c8840ca8ebd1 P 6c7c67e750b543238fa50260c28cca75: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 07:58:23.791347 20762 heartbeater.cc:503] Master 127.18.13.190:46473 was elected leader, sending a full tablet report...
I20240502 07:58:23.829991 21309 raft_consensus.cc:482] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:23.830226 21309 raft_consensus.cc:504] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a239c2e12edf43c383283f847a7a5c8e" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38353 } } peers { permanent_uuid: "fc87c2ec1ca742af906554fe00e7bbde" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 36107 } } peers { permanent_uuid: "6c7c67e750b543238fa50260c28cca75" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 45525 } }
I20240502 07:58:23.831358 21309 leader_election.cc:288] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a239c2e12edf43c383283f847a7a5c8e (127.18.13.133:38353), 6c7c67e750b543238fa50260c28cca75 (127.18.13.129:45525)
W20240502 07:58:23.833432 21025 tablet.cc:2369] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:23.835389 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4432df829b4d21a2d6c8840ca8ebd1" candidate_uuid: "fc87c2ec1ca742af906554fe00e7bbde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a239c2e12edf43c383283f847a7a5c8e" is_pre_election: true
I20240502 07:58:23.835395 20716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4432df829b4d21a2d6c8840ca8ebd1" candidate_uuid: "fc87c2ec1ca742af906554fe00e7bbde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7c67e750b543238fa50260c28cca75" is_pre_election: true
I20240502 07:58:23.835709 21240 raft_consensus.cc:2454] T 4f4432df829b4d21a2d6c8840ca8ebd1 P a239c2e12edf43c383283f847a7a5c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc87c2ec1ca742af906554fe00e7bbde in term 0.
I20240502 07:58:23.835709 20716 raft_consensus.cc:2454] T 4f4432df829b4d21a2d6c8840ca8ebd1 P 6c7c67e750b543238fa50260c28cca75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc87c2ec1ca742af906554fe00e7bbde in term 0.
I20240502 07:58:23.836244 20914 leader_election.cc:302] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c7c67e750b543238fa50260c28cca75, fc87c2ec1ca742af906554fe00e7bbde; no voters:
I20240502 07:58:23.836508 21309 raft_consensus.cc:2781] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:23.836596 21309 raft_consensus.cc:482] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:23.836637 21309 raft_consensus.cc:3037] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:23.837507 21309 raft_consensus.cc:504] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a239c2e12edf43c383283f847a7a5c8e" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38353 } } peers { permanent_uuid: "fc87c2ec1ca742af906554fe00e7bbde" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 36107 } } peers { permanent_uuid: "6c7c67e750b543238fa50260c28cca75" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 45525 } }
I20240502 07:58:23.837816 21309 leader_election.cc:288] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [CANDIDATE]: Term 1 election: Requested vote from peers a239c2e12edf43c383283f847a7a5c8e (127.18.13.133:38353), 6c7c67e750b543238fa50260c28cca75 (127.18.13.129:45525)
I20240502 07:58:23.838165 21240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4432df829b4d21a2d6c8840ca8ebd1" candidate_uuid: "fc87c2ec1ca742af906554fe00e7bbde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a239c2e12edf43c383283f847a7a5c8e"
I20240502 07:58:23.838199 20716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f4432df829b4d21a2d6c8840ca8ebd1" candidate_uuid: "fc87c2ec1ca742af906554fe00e7bbde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7c67e750b543238fa50260c28cca75"
I20240502 07:58:23.838275 21240 raft_consensus.cc:3037] T 4f4432df829b4d21a2d6c8840ca8ebd1 P a239c2e12edf43c383283f847a7a5c8e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:23.838331 20716 raft_consensus.cc:3037] T 4f4432df829b4d21a2d6c8840ca8ebd1 P 6c7c67e750b543238fa50260c28cca75 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:23.839129 21240 raft_consensus.cc:2454] T 4f4432df829b4d21a2d6c8840ca8ebd1 P a239c2e12edf43c383283f847a7a5c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc87c2ec1ca742af906554fe00e7bbde in term 1.
I20240502 07:58:23.839234 20716 raft_consensus.cc:2454] T 4f4432df829b4d21a2d6c8840ca8ebd1 P 6c7c67e750b543238fa50260c28cca75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc87c2ec1ca742af906554fe00e7bbde in term 1.
I20240502 07:58:23.839411 20911 leader_election.cc:302] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a239c2e12edf43c383283f847a7a5c8e, fc87c2ec1ca742af906554fe00e7bbde; no voters:
I20240502 07:58:23.839589 21309 raft_consensus.cc:2781] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:23.839828 21309 raft_consensus.cc:686] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [term 1 LEADER]: Becoming Leader. State: Replica: fc87c2ec1ca742af906554fe00e7bbde, State: Running, Role: LEADER
I20240502 07:58:23.840178 21309 consensus_queue.cc:237] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a239c2e12edf43c383283f847a7a5c8e" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38353 } } peers { permanent_uuid: "fc87c2ec1ca742af906554fe00e7bbde" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 36107 } } peers { permanent_uuid: "6c7c67e750b543238fa50260c28cca75" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 45525 } }
I20240502 07:58:23.843134 20573 catalog_manager.cc:5484] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde reported cstate change: term changed from 0 to 1, leader changed from <none> to fc87c2ec1ca742af906554fe00e7bbde (127.18.13.131). New cstate: current_term: 1 leader_uuid: "fc87c2ec1ca742af906554fe00e7bbde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a239c2e12edf43c383283f847a7a5c8e" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc87c2ec1ca742af906554fe00e7bbde" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 36107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c7c67e750b543238fa50260c28cca75" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 45525 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:23.864014 18486 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 07:58:23.865576 18486 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6c7c67e750b543238fa50260c28cca75 to finish bootstrapping
I20240502 07:58:23.870884 18486 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f4844e363fbb401e8185a120d15cb281 to finish bootstrapping
I20240502 07:58:23.876668 18486 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fc87c2ec1ca742af906554fe00e7bbde to finish bootstrapping
I20240502 07:58:23.881317 18486 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4a323a9af840484b803348390bbf026d to finish bootstrapping
I20240502 07:58:23.887194 18486 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a239c2e12edf43c383283f847a7a5c8e to finish bootstrapping
W20240502 07:58:23.972000 21287 tablet.cc:2369] T 4f4432df829b4d21a2d6c8840ca8ebd1 P a239c2e12edf43c383283f847a7a5c8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 07:58:23.977600 20763 tablet.cc:2369] T 4f4432df829b4d21a2d6c8840ca8ebd1 P 6c7c67e750b543238fa50260c28cca75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:24.078300 20696 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.078366 20827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.078387 21220 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.079005 21089 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.084301 20958 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:24.173151 20893 heartbeater.cc:503] Master 127.18.13.190:46473 was elected leader, sending a full tablet report...
I20240502 07:58:24.177022 20827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.177395 21220 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.178225 20958 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:24.178802 21089 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.185629 20696 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.358033 21395 consensus_queue.cc:1035] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c7c67e750b543238fa50260c28cca75" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 45525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:24.363987 21401 consensus_queue.cc:1035] T 4f4432df829b4d21a2d6c8840ca8ebd1 P fc87c2ec1ca742af906554fe00e7bbde [LEADER]: Connected to new peer: Peer: permanent_uuid: "a239c2e12edf43c383283f847a7a5c8e" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:24.405874 21089 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.406165 21220 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.407176 20696 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.407696 20958 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:24.407892 20827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.497912 21220 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.498258 21089 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.498548 20696 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.499533 20827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.510385 20958 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:24.542666 21155 heartbeater.cc:503] Master 127.18.13.190:46473 was elected leader, sending a full tablet report...
I20240502 07:58:24.600486 20827 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.600993 21089 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.601748 21220 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.604460 20696 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:24.613607 20958 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:24.641146 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 20634
I20240502 07:58:24.647001 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 20765
I20240502 07:58:24.652818 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 20896
I20240502 07:58:24.658587 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 21027
I20240502 07:58:24.664000 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 21158
I20240502 07:58:24.669409 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 20543
2024-05-02T07:58:24Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (2136 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2136 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/3
I20240502 07:58:24.692003 18486 test_util.cc:268] Using random seed: -139217123
I20240502 07:58:24.692757 18486 ts_itest-base.cc:112] Starting cluster with:
I20240502 07:58:24.692793 18486 ts_itest-base.cc:113] --------------
I20240502 07:58:24.692821 18486 ts_itest-base.cc:114] 7 tablet servers
I20240502 07:58:24.692837 18486 ts_itest-base.cc:115] 2 replicas per TS
I20240502 07:58:24.692863 18486 ts_itest-base.cc:116] --------------
2024-05-02T07:58:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T07:58:24Z Disabled control of system clock
I20240502 07:58:24.700779 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.13.190:33713
--webserver_interface=127.18.13.190
--webserver_port=0
--builtin_ntp_servers=127.18.13.148:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.13.190:33713
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 07:58:24.805127 21499 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 07:58:24.805374 21499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:24.805423 21499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:24.805497 21499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:24.808454 21499 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:24.808526 21499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 07:58:24.808566 21499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:24.808605 21499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 07:58:24.808640 21499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 07:58:24.812176 21499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.18.13.190:33713
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.13.190:33713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.18.13.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.21499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:24.813027 21499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:24.814118 21499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:24.822197 21507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:24.823328 21505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:24.823884 21504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:24.825006 21499 server_base.cc:1034] running on GCE node
I20240502 07:58:24.825496 21499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:24.826404 21499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:24.827553 21499 hybrid_clock.cc:648] HybridClock initialized: now 1714636704827542 us; error 26 us; skew 500 ppm
I20240502 07:58:24.833735 21499 webserver.cc:460] Webserver started at http://127.18.13.190:33523/ using document root <none> and password file <none>
I20240502 07:58:24.834301 21499 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:24.834362 21499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:24.834545 21499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:24.836313 21499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "146dd163130c4e40ab0cc570b6cc2f0e"
format_stamp: "Formatted at 2024-05-02 07:58:24 on dist-test-slave-f46z"
I20240502 07:58:24.836795 21499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "146dd163130c4e40ab0cc570b6cc2f0e"
format_stamp: "Formatted at 2024-05-02 07:58:24 on dist-test-slave-f46z"
I20240502 07:58:24.840029 21499 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 07:58:24.842140 21513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:24.843083 21499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:24.843194 21499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
uuid: "146dd163130c4e40ab0cc570b6cc2f0e"
format_stamp: "Formatted at 2024-05-02 07:58:24 on dist-test-slave-f46z"
I20240502 07:58:24.843274 21499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:24.861766 21499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:24.862350 21499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:24.862504 21499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:24.868786 21499 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.190:33713
I20240502 07:58:24.868795 21565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.190:33713 every 8 connection(s)
I20240502 07:58:24.869679 21499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 07:58:24.872007 21566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:24.875856 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 21499
I20240502 07:58:24.875975 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 07:58:24.877348 21566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e: Bootstrap starting.
I20240502 07:58:24.879370 21566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:24.880129 21566 log.cc:824] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:24.881567 21566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e: No bootstrap required, opened a new log
I20240502 07:58:24.884093 21566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146dd163130c4e40ab0cc570b6cc2f0e" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 33713 } }
I20240502 07:58:24.884271 21566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:24.884354 21566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146dd163130c4e40ab0cc570b6cc2f0e, State: Initialized, Role: FOLLOWER
I20240502 07:58:24.884773 21566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146dd163130c4e40ab0cc570b6cc2f0e" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 33713 } }
I20240502 07:58:24.884910 21566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:24.884974 21566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:24.885069 21566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:24.885733 21566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146dd163130c4e40ab0cc570b6cc2f0e" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 33713 } }
I20240502 07:58:24.886018 21566 leader_election.cc:302] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 146dd163130c4e40ab0cc570b6cc2f0e; no voters:
I20240502 07:58:24.886252 21566 leader_election.cc:288] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:58:24.886560 21571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:24.886734 21571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [term 1 LEADER]: Becoming Leader. State: Replica: 146dd163130c4e40ab0cc570b6cc2f0e, State: Running, Role: LEADER
I20240502 07:58:24.887033 21571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146dd163130c4e40ab0cc570b6cc2f0e" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 33713 } }
I20240502 07:58:24.887233 21566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:58:24.888330 21572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "146dd163130c4e40ab0cc570b6cc2f0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146dd163130c4e40ab0cc570b6cc2f0e" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 33713 } } }
I20240502 07:58:24.888425 21572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:24.888726 21573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 146dd163130c4e40ab0cc570b6cc2f0e. Latest consensus state: current_term: 1 leader_uuid: "146dd163130c4e40ab0cc570b6cc2f0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "146dd163130c4e40ab0cc570b6cc2f0e" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 33713 } } }
I20240502 07:58:24.888833 21573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:24.888856 21580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:58:24.891754 21580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:58:24.896204 21580 catalog_manager.cc:1347] Generated new cluster ID: cca4f8c5f85b4ed083f5799f0a53a84d
I20240502 07:58:24.896267 21580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:58:24.904963 21580 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:58:24.905736 21580 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:58:24.916424 21580 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 146dd163130c4e40ab0cc570b6cc2f0e: Generated new TSK 0
I20240502 07:58:24.917055 21580 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:58:24.926931 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.129:0
--local_ip_for_outbound_sockets=127.18.13.129
--webserver_interface=127.18.13.129
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--builtin_ntp_servers=127.18.13.148:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 07:58:25.032385 21590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:25.032588 21590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:25.032660 21590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:25.035605 21590 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:25.035678 21590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:25.035768 21590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.129
I20240502 07:58:25.039297 21590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.18.13.129
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--never_fsync=true
--heap_profile_path=/tmp/kudu.21590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.129
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:25.040184 21590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:25.041286 21590 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:25.049731 21598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.051005 21595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.051923 21596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:25.053211 21590 server_base.cc:1034] running on GCE node
I20240502 07:58:25.053575 21590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:25.054059 21590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:25.055203 21590 hybrid_clock.cc:648] HybridClock initialized: now 1714636705055181 us; error 36 us; skew 500 ppm
I20240502 07:58:25.056872 21590 webserver.cc:460] Webserver started at http://127.18.13.129:37701/ using document root <none> and password file <none>
I20240502 07:58:25.057374 21590 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:25.057426 21590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:25.057577 21590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:25.059120 21590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e218451f770e4ab0a5266e60bdb30a6a"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.059542 21590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e218451f770e4ab0a5266e60bdb30a6a"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.062677 21590 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240502 07:58:25.064687 21604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.065610 21590 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:25.065712 21590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e218451f770e4ab0a5266e60bdb30a6a"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.065791 21590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:25.089187 21590 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:25.089833 21590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:25.089998 21590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:25.090516 21590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:25.091389 21590 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:25.091439 21590 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.091493 21590 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:25.091521 21590 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.100299 21590 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.129:40667
I20240502 07:58:25.100322 21717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.129:40667 every 8 connection(s)
I20240502 07:58:25.101207 21590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 07:58:25.102890 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 21590
I20240502 07:58:25.102988 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 07:58:25.105024 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.130:0
--local_ip_for_outbound_sockets=127.18.13.130
--webserver_interface=127.18.13.130
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--builtin_ntp_servers=127.18.13.148:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:25.110025 21718 heartbeater.cc:347] Connected to a master server at 127.18.13.190:33713
I20240502 07:58:25.110289 21718 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:25.110975 21718 heartbeater.cc:511] Master 127.18.13.190:33713 requested a full tablet report, sending...
I20240502 07:58:25.112481 21530 ts_manager.cc:194] Registered new tserver with Master: e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:25.113580 21530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.129:44887
W20240502 07:58:25.209040 21721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:25.209244 21721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:25.209317 21721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:25.212230 21721 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:25.212301 21721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:25.212397 21721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.130
I20240502 07:58:25.215864 21721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.18.13.130
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--never_fsync=true
--heap_profile_path=/tmp/kudu.21721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.130
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:25.216714 21721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:25.217798 21721 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:25.226423 21727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.227185 21726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.227888 21729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:25.229123 21721 server_base.cc:1034] running on GCE node
I20240502 07:58:25.229488 21721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:25.229986 21721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:25.231123 21721 hybrid_clock.cc:648] HybridClock initialized: now 1714636705231087 us; error 51 us; skew 500 ppm
I20240502 07:58:25.232854 21721 webserver.cc:460] Webserver started at http://127.18.13.130:46215/ using document root <none> and password file <none>
I20240502 07:58:25.233357 21721 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:25.233409 21721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:25.233549 21721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:25.235126 21721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e8406d5a62054258be7fff420a4a4fe2"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.235544 21721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e8406d5a62054258be7fff420a4a4fe2"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.238669 21721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:58:25.240658 21735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.241588 21721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:25.241676 21721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e8406d5a62054258be7fff420a4a4fe2"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.241744 21721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:25.261075 21721 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:25.261646 21721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:25.261776 21721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:25.262245 21721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:25.263067 21721 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:25.263113 21721 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.263149 21721 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:25.263167 21721 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.272023 21721 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.130:40891
I20240502 07:58:25.272078 21848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.130:40891 every 8 connection(s)
I20240502 07:58:25.272910 21721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 07:58:25.280364 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 21721
I20240502 07:58:25.280475 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 07:58:25.281755 21849 heartbeater.cc:347] Connected to a master server at 127.18.13.190:33713
I20240502 07:58:25.281996 21849 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:25.282524 21849 heartbeater.cc:511] Master 127.18.13.190:33713 requested a full tablet report, sending...
I20240502 07:58:25.282824 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.131:0
--local_ip_for_outbound_sockets=127.18.13.131
--webserver_interface=127.18.13.131
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--builtin_ntp_servers=127.18.13.148:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:25.283577 21530 ts_manager.cc:194] Registered new tserver with Master: e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:25.284399 21530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.130:39357
W20240502 07:58:25.389007 21852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:25.389245 21852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:25.389287 21852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:25.392416 21852 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:25.392496 21852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:25.392598 21852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.131
I20240502 07:58:25.396153 21852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.18.13.131
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--never_fsync=true
--heap_profile_path=/tmp/kudu.21852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.131
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:25.397020 21852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:25.398128 21852 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:25.406221 21860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.407323 21857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.408309 21858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:25.409493 21852 server_base.cc:1034] running on GCE node
I20240502 07:58:25.409844 21852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:25.410380 21852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:25.411567 21852 hybrid_clock.cc:648] HybridClock initialized: now 1714636705411544 us; error 28 us; skew 500 ppm
I20240502 07:58:25.413456 21852 webserver.cc:460] Webserver started at http://127.18.13.131:34275/ using document root <none> and password file <none>
I20240502 07:58:25.413993 21852 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:25.414052 21852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:25.414240 21852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:25.415966 21852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "21b4d9bd384846b2ad74c199585018ba"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.416441 21852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "21b4d9bd384846b2ad74c199585018ba"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.419678 21852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 07:58:25.421793 21866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.422748 21852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:25.422840 21852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "21b4d9bd384846b2ad74c199585018ba"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.422938 21852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:25.463984 21852 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:25.464653 21852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:25.464812 21852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:25.465350 21852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:25.466228 21852 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:25.466275 21852 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.466324 21852 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:25.466357 21852 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.474989 21852 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.131:38811
I20240502 07:58:25.475021 21979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.131:38811 every 8 connection(s)
I20240502 07:58:25.476083 21852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 07:58:25.478986 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 21852
I20240502 07:58:25.479082 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 07:58:25.481096 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.132:0
--local_ip_for_outbound_sockets=127.18.13.132
--webserver_interface=127.18.13.132
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--builtin_ntp_servers=127.18.13.148:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:25.485037 21980 heartbeater.cc:347] Connected to a master server at 127.18.13.190:33713
I20240502 07:58:25.485247 21980 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:25.485738 21980 heartbeater.cc:511] Master 127.18.13.190:33713 requested a full tablet report, sending...
I20240502 07:58:25.486660 21530 ts_manager.cc:194] Registered new tserver with Master: 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:25.487169 21530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.131:44203
W20240502 07:58:25.589764 21983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:25.590003 21983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:25.590075 21983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:25.593173 21983 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:25.593246 21983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:25.593338 21983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.132
I20240502 07:58:25.596968 21983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.18.13.132
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--never_fsync=true
--heap_profile_path=/tmp/kudu.21983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.132
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:25.597865 21983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:25.598997 21983 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:25.607970 21991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.609025 21988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.610157 21989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:25.611289 21983 server_base.cc:1034] running on GCE node
I20240502 07:58:25.611696 21983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:25.612326 21983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:25.613479 21983 hybrid_clock.cc:648] HybridClock initialized: now 1714636705613466 us; error 27 us; skew 500 ppm
I20240502 07:58:25.615442 21983 webserver.cc:460] Webserver started at http://127.18.13.132:44579/ using document root <none> and password file <none>
I20240502 07:58:25.616039 21983 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:25.616097 21983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:25.616284 21983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:25.618011 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ec7796d40c8a463caa442aa67ba3d367"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.618491 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ec7796d40c8a463caa442aa67ba3d367"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.621873 21983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:58:25.624109 21997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.625175 21983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 07:58:25.625291 21983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ec7796d40c8a463caa442aa67ba3d367"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.625383 21983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:25.640635 21983 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:25.641254 21983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:25.641417 21983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:25.641943 21983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:25.642783 21983 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:25.642830 21983 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.642897 21983 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:25.642936 21983 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.651716 21983 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.132:40025
I20240502 07:58:25.651739 22110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.132:40025 every 8 connection(s)
I20240502 07:58:25.652683 21983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:58:25.657531 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 21983
I20240502 07:58:25.657639 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 07:58:25.659641 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.133:0
--local_ip_for_outbound_sockets=127.18.13.133
--webserver_interface=127.18.13.133
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--builtin_ntp_servers=127.18.13.148:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:25.661098 22111 heartbeater.cc:347] Connected to a master server at 127.18.13.190:33713
I20240502 07:58:25.661337 22111 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:25.661932 22111 heartbeater.cc:511] Master 127.18.13.190:33713 requested a full tablet report, sending...
I20240502 07:58:25.662873 21530 ts_manager.cc:194] Registered new tserver with Master: ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132:40025)
I20240502 07:58:25.663643 21530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.132:49909
W20240502 07:58:25.766047 22114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:25.766256 22114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:25.766299 22114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:25.769366 22114 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:25.769438 22114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:25.769506 22114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.133
I20240502 07:58:25.773138 22114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.18.13.133
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--never_fsync=true
--heap_profile_path=/tmp/kudu.22114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.133
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:25.774017 22114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:25.775167 22114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:25.783962 22119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.784891 22122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.785749 22120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:25.787149 22114 server_base.cc:1034] running on GCE node
I20240502 07:58:25.787475 22114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:25.788045 22114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:25.789194 22114 hybrid_clock.cc:648] HybridClock initialized: now 1714636705789178 us; error 40 us; skew 500 ppm
I20240502 07:58:25.791028 22114 webserver.cc:460] Webserver started at http://127.18.13.133:36873/ using document root <none> and password file <none>
I20240502 07:58:25.791574 22114 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:25.791636 22114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:25.791842 22114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:25.793592 22114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f6079903ad93465c8741bf16583fe8c8"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.794060 22114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f6079903ad93465c8741bf16583fe8c8"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.797366 22114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 07:58:25.799454 22128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.800467 22114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:25.800595 22114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f6079903ad93465c8741bf16583fe8c8"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.800673 22114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:25.811043 22114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:25.811602 22114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:25.811728 22114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:25.812203 22114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:25.812994 22114 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:25.813038 22114 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.813076 22114 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:25.813095 22114 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.821444 22114 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.133:38189
I20240502 07:58:25.821462 22241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.133:38189 every 8 connection(s)
I20240502 07:58:25.822340 22114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:58:25.825587 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 22114
I20240502 07:58:25.825698 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 07:58:25.827749 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.134:0
--local_ip_for_outbound_sockets=127.18.13.134
--webserver_interface=127.18.13.134
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--builtin_ntp_servers=127.18.13.148:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:25.831491 22242 heartbeater.cc:347] Connected to a master server at 127.18.13.190:33713
I20240502 07:58:25.831709 22242 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:25.832266 22242 heartbeater.cc:511] Master 127.18.13.190:33713 requested a full tablet report, sending...
I20240502 07:58:25.833307 21530 ts_manager.cc:194] Registered new tserver with Master: f6079903ad93465c8741bf16583fe8c8 (127.18.13.133:38189)
I20240502 07:58:25.834116 21530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.133:53035
W20240502 07:58:25.936235 22245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:25.936452 22245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:25.936496 22245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:25.939427 22245 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:25.939489 22245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:25.939555 22245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.134
I20240502 07:58:25.943070 22245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.18.13.134
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--never_fsync=true
--heap_profile_path=/tmp/kudu.22245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.134
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:25.943957 22245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:25.945104 22245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:25.955006 22253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.955946 22251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:25.956840 22250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:25.958338 22245 server_base.cc:1034] running on GCE node
I20240502 07:58:25.958922 22245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:25.959729 22245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:25.960943 22245 hybrid_clock.cc:648] HybridClock initialized: now 1714636705960912 us; error 47 us; skew 500 ppm
I20240502 07:58:25.962838 22245 webserver.cc:460] Webserver started at http://127.18.13.134:44811/ using document root <none> and password file <none>
I20240502 07:58:25.963424 22245 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:25.963482 22245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:25.963655 22245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:25.965381 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.965842 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.970438 22245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 07:58:25.973376 22259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.974393 22245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:25.974503 22245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:25.974593 22245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:25.989830 22245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:25.990432 22245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:25.990581 22245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:25.991091 22245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:25.991979 22245 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:25.992028 22245 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:25.992082 22245 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:25.992112 22245 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:26.000751 22245 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.134:42113
I20240502 07:58:26.000774 22372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.134:42113 every 8 connection(s)
I20240502 07:58:26.001672 22245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 07:58:26.003504 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 22245
I20240502 07:58:26.003646 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 07:58:26.006417 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.135:0
--local_ip_for_outbound_sockets=127.18.13.135
--webserver_interface=127.18.13.135
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--builtin_ntp_servers=127.18.13.148:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:26.011425 22373 heartbeater.cc:347] Connected to a master server at 127.18.13.190:33713
I20240502 07:58:26.011643 22373 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:26.012307 22373 heartbeater.cc:511] Master 127.18.13.190:33713 requested a full tablet report, sending...
I20240502 07:58:26.013258 21530 ts_manager.cc:194] Registered new tserver with Master: c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134:42113)
I20240502 07:58:26.013784 21530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.134:60329
W20240502 07:58:26.115959 22376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:26.116205 22376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20240502 07:58:26.116182 21718 heartbeater.cc:503] Master 127.18.13.190:33713 was elected leader, sending a full tablet report...
W20240502 07:58:26.116284 22376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:26.119505 22376 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:26.119580 22376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:26.119684 22376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.135
I20240502 07:58:26.123523 22376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.18.13.135
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--never_fsync=true
--heap_profile_path=/tmp/kudu.22376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.135
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:26.124502 22376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:26.125605 22376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:26.134652 22382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:26.135725 22381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:26.136687 22384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:26.137593 22376 server_base.cc:1034] running on GCE node
I20240502 07:58:26.137974 22376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:26.138556 22376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:26.139722 22376 hybrid_clock.cc:648] HybridClock initialized: now 1714636706139706 us; error 28 us; skew 500 ppm
I20240502 07:58:26.141573 22376 webserver.cc:460] Webserver started at http://127.18.13.135:36617/ using document root <none> and password file <none>
I20240502 07:58:26.142130 22376 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:26.142192 22376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:26.142370 22376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:26.144116 22376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "f97f8632bbe64638bb1ad2679849f434"
format_stamp: "Formatted at 2024-05-02 07:58:26 on dist-test-slave-f46z"
I20240502 07:58:26.144575 22376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "f97f8632bbe64638bb1ad2679849f434"
format_stamp: "Formatted at 2024-05-02 07:58:26 on dist-test-slave-f46z"
I20240502 07:58:26.147933 22376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 07:58:26.149989 22390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:26.151017 22376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:26.151156 22376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f97f8632bbe64638bb1ad2679849f434"
format_stamp: "Formatted at 2024-05-02 07:58:26 on dist-test-slave-f46z"
I20240502 07:58:26.151257 22376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:26.162909 22376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:26.163497 22376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:26.163650 22376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:26.164304 22376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:26.165179 22376 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:26.165225 22376 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:26.165263 22376 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:26.165300 22376 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:26.173488 22376 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.135:43407
I20240502 07:58:26.173529 22503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.135:43407 every 8 connection(s)
I20240502 07:58:26.174386 22376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 07:58:26.182698 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 22376
I20240502 07:58:26.182834 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 07:58:26.183172 22504 heartbeater.cc:347] Connected to a master server at 127.18.13.190:33713
I20240502 07:58:26.183406 22504 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:26.184178 22504 heartbeater.cc:511] Master 127.18.13.190:33713 requested a full tablet report, sending...
I20240502 07:58:26.185209 21530 ts_manager.cc:194] Registered new tserver with Master: f97f8632bbe64638bb1ad2679849f434 (127.18.13.135:43407)
I20240502 07:58:26.185716 21530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.135:60637
I20240502 07:58:26.196902 18486 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 07:58:26.205358 21530 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 07:58:26.221923 21914 tablet_service.cc:1467] Processing CreateTablet for tablet 01e44d65c0314f0a9d30a2587723c290 (DEFAULT_TABLE table=TestTable [id=828c706eb9dd489bb081a01b553ce7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:26.221923 21783 tablet_service.cc:1467] Processing CreateTablet for tablet 01e44d65c0314f0a9d30a2587723c290 (DEFAULT_TABLE table=TestTable [id=828c706eb9dd489bb081a01b553ce7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:26.222921 21783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e44d65c0314f0a9d30a2587723c290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:26.222921 21914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e44d65c0314f0a9d30a2587723c290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:26.228873 22521 tablet_bootstrap.cc:492] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba: Bootstrap starting.
I20240502 07:58:26.229933 22520 tablet_bootstrap.cc:492] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2: Bootstrap starting.
I20240502 07:58:26.231796 22520 tablet_bootstrap.cc:654] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:26.232645 22520 log.cc:824] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:26.233125 22521 tablet_bootstrap.cc:654] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:26.233716 22521 log.cc:824] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:26.234244 22520 tablet_bootstrap.cc:492] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2: No bootstrap required, opened a new log
I20240502 07:58:26.234419 22520 ts_tablet_manager.cc:1367] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 07:58:26.235105 22521 tablet_bootstrap.cc:492] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba: No bootstrap required, opened a new log
I20240502 07:58:26.235255 22521 ts_tablet_manager.cc:1367] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 07:58:26.236873 22520 raft_consensus.cc:348] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:26.237107 22520 raft_consensus.cc:374] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:26.237164 22520 raft_consensus.cc:729] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Initialized, Role: FOLLOWER
I20240502 07:58:26.237605 22520 consensus_queue.cc:260] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:26.237773 22521 raft_consensus.cc:348] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:26.237949 22521 raft_consensus.cc:374] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:26.238010 22521 raft_consensus.cc:729] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Initialized, Role: FOLLOWER
I20240502 07:58:26.238435 22520 ts_tablet_manager.cc:1398] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20240502 07:58:26.238399 22521 consensus_queue.cc:260] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:26.238446 21849 heartbeater.cc:503] Master 127.18.13.190:33713 was elected leader, sending a full tablet report...
I20240502 07:58:26.239137 22521 ts_tablet_manager.cc:1398] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 07:58:26.239146 21980 heartbeater.cc:503] Master 127.18.13.190:33713 was elected leader, sending a full tablet report...
W20240502 07:58:26.274449 21850 tablet.cc:2369] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:26.413998 22525 raft_consensus.cc:482] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:26.414204 22525 raft_consensus.cc:504] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:26.415278 22525 leader_election.cc:288] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:26.418669 21803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e44d65c0314f0a9d30a2587723c290" candidate_uuid: "21b4d9bd384846b2ad74c199585018ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8406d5a62054258be7fff420a4a4fe2" is_pre_election: true
I20240502 07:58:26.418951 21803 raft_consensus.cc:2454] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21b4d9bd384846b2ad74c199585018ba in term 0.
I20240502 07:58:26.419384 21868 leader_election.cc:302] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:26.419610 22525 raft_consensus.cc:2781] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:26.419683 22525 raft_consensus.cc:482] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:26.419734 22525 raft_consensus.cc:3037] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:26.420603 22525 raft_consensus.cc:504] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:26.420871 22525 leader_election.cc:288] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 election: Requested vote from peers e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:26.421234 21803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e44d65c0314f0a9d30a2587723c290" candidate_uuid: "21b4d9bd384846b2ad74c199585018ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
I20240502 07:58:26.421336 21803 raft_consensus.cc:3037] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:26.422143 21803 raft_consensus.cc:2454] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21b4d9bd384846b2ad74c199585018ba in term 1.
I20240502 07:58:26.422415 21868 leader_election.cc:302] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:26.422569 22525 raft_consensus.cc:2781] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:26.422776 22525 raft_consensus.cc:686] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Becoming Leader. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Running, Role: LEADER
I20240502 07:58:26.423089 22525 consensus_queue.cc:237] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:26.425971 21529 catalog_manager.cc:5484] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 21b4d9bd384846b2ad74c199585018ba (127.18.13.131). New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:26.450193 18486 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 07:58:26.451618 18486 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e218451f770e4ab0a5266e60bdb30a6a to finish bootstrapping
I20240502 07:58:26.457231 18486 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e8406d5a62054258be7fff420a4a4fe2 to finish bootstrapping
I20240502 07:58:26.461742 18486 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 21b4d9bd384846b2ad74c199585018ba to finish bootstrapping
I20240502 07:58:26.466187 18486 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ec7796d40c8a463caa442aa67ba3d367 to finish bootstrapping
I20240502 07:58:26.471545 18486 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f6079903ad93465c8741bf16583fe8c8 to finish bootstrapping
I20240502 07:58:26.476716 18486 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c0ee0b72cadc4ebaa2e32a046a107ce6 to finish bootstrapping
W20240502 07:58:26.477522 21981 tablet.cc:2369] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:26.484892 18486 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f97f8632bbe64638bb1ad2679849f434 to finish bootstrapping
I20240502 07:58:26.489953 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 21983
I20240502 07:58:26.495517 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 22114
I20240502 07:58:26.501268 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 22245
I20240502 07:58:26.506559 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 22376
I20240502 07:58:27.032065 22535 consensus_queue.cc:1035] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:30.263156 21530 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:30.270324 21783 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed04eb8c46f4cf4bf721d75baa8d534 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b1571debd96c4d31a1be20579ff6bb36]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:30.270642 21783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed04eb8c46f4cf4bf721d75baa8d534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:30.270828 21782 tablet_service.cc:1467] Processing CreateTablet for tablet 0029538492ea4f25a822f18de9c83c54 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b1571debd96c4d31a1be20579ff6bb36]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:30.271090 21782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0029538492ea4f25a822f18de9c83c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:30.271034 21914 tablet_service.cc:1467] Processing CreateTablet for tablet f3f58339fd2c4528a2decb910ea73e21 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b1571debd96c4d31a1be20579ff6bb36]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:30.271293 21914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f58339fd2c4528a2decb910ea73e21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:30.274057 22547 tablet_bootstrap.cc:492] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2: Bootstrap starting.
I20240502 07:58:30.274269 22549 tablet_bootstrap.cc:492] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba: Bootstrap starting.
I20240502 07:58:30.275081 22547 tablet_bootstrap.cc:654] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:30.275243 22549 tablet_bootstrap.cc:654] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:30.276386 22549 tablet_bootstrap.cc:492] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba: No bootstrap required, opened a new log
I20240502 07:58:30.276484 22549 ts_tablet_manager.cc:1367] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:30.277089 22549 raft_consensus.cc:348] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:30.277185 22547 tablet_bootstrap.cc:492] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2: No bootstrap required, opened a new log
I20240502 07:58:30.277196 22549 raft_consensus.cc:374] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:30.277227 22549 raft_consensus.cc:729] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Initialized, Role: FOLLOWER
I20240502 07:58:30.277254 22547 ts_tablet_manager.cc:1367] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:30.277364 22549 consensus_queue.cc:260] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:30.277652 22549 ts_tablet_manager.cc:1398] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:30.277771 22547 raft_consensus.cc:348] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:30.277871 22547 raft_consensus.cc:374] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:30.277902 22547 raft_consensus.cc:729] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Initialized, Role: FOLLOWER
I20240502 07:58:30.278035 22547 consensus_queue.cc:260] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:30.278337 22547 ts_tablet_manager.cc:1398] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:30.278493 22547 tablet_bootstrap.cc:492] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2: Bootstrap starting.
I20240502 07:58:30.279388 22547 tablet_bootstrap.cc:654] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:30.280395 21652 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed04eb8c46f4cf4bf721d75baa8d534 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b1571debd96c4d31a1be20579ff6bb36]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:30.281464 21652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed04eb8c46f4cf4bf721d75baa8d534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:30.280697 21650 tablet_service.cc:1467] Processing CreateTablet for tablet 0029538492ea4f25a822f18de9c83c54 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b1571debd96c4d31a1be20579ff6bb36]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:30.282402 21650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0029538492ea4f25a822f18de9c83c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:30.282514 22547 tablet_bootstrap.cc:492] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2: No bootstrap required, opened a new log
I20240502 07:58:30.282590 22547 ts_tablet_manager.cc:1367] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 07:58:30.283035 22547 raft_consensus.cc:348] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:30.283142 22547 raft_consensus.cc:374] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:30.283177 22547 raft_consensus.cc:729] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Initialized, Role: FOLLOWER
I20240502 07:58:30.283290 22547 consensus_queue.cc:260] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:30.283567 22547 ts_tablet_manager.cc:1398] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:30.280407 21651 tablet_service.cc:1467] Processing CreateTablet for tablet f3f58339fd2c4528a2decb910ea73e21 (DEFAULT_TABLE table=rebalance_test_table_0 [id=b1571debd96c4d31a1be20579ff6bb36]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:30.284930 21651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f58339fd2c4528a2decb910ea73e21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:30.288079 22554 tablet_bootstrap.cc:492] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a: Bootstrap starting.
I20240502 07:58:30.289947 22554 tablet_bootstrap.cc:654] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:30.290791 22554 log.cc:824] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:30.292337 22554 tablet_bootstrap.cc:492] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a: No bootstrap required, opened a new log
I20240502 07:58:30.292480 22554 ts_tablet_manager.cc:1367] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 07:58:30.295137 22554 raft_consensus.cc:348] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:30.295333 22554 raft_consensus.cc:374] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:30.295394 22554 raft_consensus.cc:729] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Initialized, Role: FOLLOWER
I20240502 07:58:30.295830 22554 consensus_queue.cc:260] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:30.296586 22554 ts_tablet_manager.cc:1398] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 07:58:30.296883 22554 tablet_bootstrap.cc:492] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a: Bootstrap starting.
I20240502 07:58:30.297745 22554 tablet_bootstrap.cc:654] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:30.298949 22554 tablet_bootstrap.cc:492] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a: No bootstrap required, opened a new log
I20240502 07:58:30.299062 22554 ts_tablet_manager.cc:1367] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:30.299633 22554 raft_consensus.cc:348] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:30.299727 22554 raft_consensus.cc:374] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:30.299763 22554 raft_consensus.cc:729] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Initialized, Role: FOLLOWER
I20240502 07:58:30.299927 22554 consensus_queue.cc:260] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:30.300204 22554 ts_tablet_manager.cc:1398] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:30.300352 22554 tablet_bootstrap.cc:492] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a: Bootstrap starting.
I20240502 07:58:30.301177 22554 tablet_bootstrap.cc:654] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:30.302315 22554 tablet_bootstrap.cc:492] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a: No bootstrap required, opened a new log
I20240502 07:58:30.302405 22554 ts_tablet_manager.cc:1367] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:58:30.302772 22554 raft_consensus.cc:348] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:30.302855 22554 raft_consensus.cc:374] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:30.302875 22554 raft_consensus.cc:729] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Initialized, Role: FOLLOWER
I20240502 07:58:30.302991 22554 consensus_queue.cc:260] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:30.303246 22554 ts_tablet_manager.cc:1398] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:30.336544 22552 raft_consensus.cc:482] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:30.336692 22552 raft_consensus.cc:504] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:30.337622 22552 leader_election.cc:288] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:30.341841 21672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed04eb8c46f4cf4bf721d75baa8d534" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a" is_pre_election: true
I20240502 07:58:30.342128 21672 raft_consensus.cc:2454] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 0.
I20240502 07:58:30.342523 21737 leader_election.cc:302] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e218451f770e4ab0a5266e60bdb30a6a, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:30.342767 22552 raft_consensus.cc:2781] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:30.342823 22552 raft_consensus.cc:482] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:30.342849 22552 raft_consensus.cc:3037] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:30.343694 22552 raft_consensus.cc:504] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:30.343976 22552 leader_election.cc:288] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 election: Requested vote from peers e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:30.344295 21672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed04eb8c46f4cf4bf721d75baa8d534" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a"
I20240502 07:58:30.344396 21672 raft_consensus.cc:3037] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:30.345189 21672 raft_consensus.cc:2454] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 1.
I20240502 07:58:30.345430 21737 leader_election.cc:302] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e218451f770e4ab0a5266e60bdb30a6a, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:30.345615 22552 raft_consensus.cc:2781] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:30.345842 22552 raft_consensus.cc:686] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Becoming Leader. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Running, Role: LEADER
I20240502 07:58:30.346139 22552 consensus_queue.cc:237] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:30.348114 21530 catalog_manager.cc:5484] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:30.356050 21719 tablet.cc:2369] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:30.726130 22552 raft_consensus.cc:482] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:30.726269 22552 raft_consensus.cc:504] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:30.726650 22552 leader_election.cc:288] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:30.727161 21672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0029538492ea4f25a822f18de9c83c54" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a" is_pre_election: true
I20240502 07:58:30.727293 21672 raft_consensus.cc:2454] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 0.
I20240502 07:58:30.727587 21737 leader_election.cc:302] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e218451f770e4ab0a5266e60bdb30a6a, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:30.727815 22552 raft_consensus.cc:2781] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:30.727876 22552 raft_consensus.cc:482] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:30.727905 22552 raft_consensus.cc:3037] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:30.728772 22552 raft_consensus.cc:504] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:30.729048 22552 leader_election.cc:288] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 election: Requested vote from peers e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:30.729416 21672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0029538492ea4f25a822f18de9c83c54" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a"
I20240502 07:58:30.729523 21672 raft_consensus.cc:3037] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:30.730244 21672 raft_consensus.cc:2454] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 1.
I20240502 07:58:30.730522 21737 leader_election.cc:302] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e218451f770e4ab0a5266e60bdb30a6a, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:30.730692 22552 raft_consensus.cc:2781] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:30.730777 22552 raft_consensus.cc:686] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Becoming Leader. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Running, Role: LEADER
I20240502 07:58:30.730916 22552 consensus_queue.cc:237] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:30.732378 21530 catalog_manager.cc:5484] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:30.755962 22541 raft_consensus.cc:482] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:30.756107 22541 raft_consensus.cc:504] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:30.756572 22541 leader_election.cc:288] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:30.759832 21672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f58339fd2c4528a2decb910ea73e21" candidate_uuid: "21b4d9bd384846b2ad74c199585018ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a" is_pre_election: true
I20240502 07:58:30.759984 21672 raft_consensus.cc:2454] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21b4d9bd384846b2ad74c199585018ba in term 0.
I20240502 07:58:30.760288 21868 leader_election.cc:302] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e218451f770e4ab0a5266e60bdb30a6a; no voters:
I20240502 07:58:30.760465 22541 raft_consensus.cc:2781] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:30.760541 22541 raft_consensus.cc:482] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:30.760569 22541 raft_consensus.cc:3037] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:30.761343 22541 raft_consensus.cc:504] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:30.761615 22541 leader_election.cc:288] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 election: Requested vote from peers e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:30.761983 21672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3f58339fd2c4528a2decb910ea73e21" candidate_uuid: "21b4d9bd384846b2ad74c199585018ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a"
I20240502 07:58:30.762102 21672 raft_consensus.cc:3037] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:30.762840 21672 raft_consensus.cc:2454] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21b4d9bd384846b2ad74c199585018ba in term 1.
I20240502 07:58:30.763120 21868 leader_election.cc:302] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e218451f770e4ab0a5266e60bdb30a6a; no voters:
I20240502 07:58:30.763271 22541 raft_consensus.cc:2781] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:30.763425 22541 raft_consensus.cc:686] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Becoming Leader. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Running, Role: LEADER
I20240502 07:58:30.763578 22541 consensus_queue.cc:237] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:30.765365 21530 catalog_manager.cc:5484] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 21b4d9bd384846b2ad74c199585018ba (127.18.13.131). New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:30.823009 22552 consensus_queue.cc:1035] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 07:58:30.942783 22570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:30.943009 22570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:30.945961 22570 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 07:58:31.042246 21671 raft_consensus.cc:1261] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:31.042862 22541 consensus_queue.cc:1035] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:31.047183 21670 raft_consensus.cc:1261] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:31.047827 22560 consensus_queue.cc:1035] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:31.056950 22601 mvcc.cc:205] Tried to move back new op lower bound from 7023151968420360192 to 7023151967288717312. Current Snapshot: MvccSnapshot[applied={T|T < 7023151968420360192}]
I20240502 07:58:31.062072 22607 mvcc.cc:205] Tried to move back new op lower bound from 7023151968416538624 to 7023151967154606080. Current Snapshot: MvccSnapshot[applied={T|T < 7023151968416538624}]
INSERT report
rows total: 2000
time total: 95.6591 ms
time per row: 0.0478296 ms
I20240502 07:58:31.088726 21530 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:31.094537 21652 tablet_service.cc:1467] Processing CreateTablet for tablet e46a300254bb45d1970dfb5db271ad14 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5d61f2326983407fa0ce10f3bd2dcf48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:31.094575 21914 tablet_service.cc:1467] Processing CreateTablet for tablet e46a300254bb45d1970dfb5db271ad14 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5d61f2326983407fa0ce10f3bd2dcf48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:31.094882 21652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46a300254bb45d1970dfb5db271ad14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:31.094899 21914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46a300254bb45d1970dfb5db271ad14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:31.096135 21912 tablet_service.cc:1467] Processing CreateTablet for tablet 4b138b61a1604cd18fe31b0f4ffb28d2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5d61f2326983407fa0ce10f3bd2dcf48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:31.096391 21912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b138b61a1604cd18fe31b0f4ffb28d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:31.097007 22615 tablet_bootstrap.cc:492] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a: Bootstrap starting.
I20240502 07:58:31.097787 22615 tablet_bootstrap.cc:654] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:31.098294 21913 tablet_service.cc:1467] Processing CreateTablet for tablet c725ad05375344c3b52682bc91090a43 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5d61f2326983407fa0ce10f3bd2dcf48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:31.098544 21913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c725ad05375344c3b52682bc91090a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:31.100550 21652 tablet_service.cc:1467] Processing CreateTablet for tablet 4b138b61a1604cd18fe31b0f4ffb28d2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5d61f2326983407fa0ce10f3bd2dcf48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:31.100555 21782 tablet_service.cc:1467] Processing CreateTablet for tablet c725ad05375344c3b52682bc91090a43 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5d61f2326983407fa0ce10f3bd2dcf48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:31.100831 21782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c725ad05375344c3b52682bc91090a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:31.100957 22616 tablet_bootstrap.cc:492] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba: Bootstrap starting.
I20240502 07:58:31.101078 21652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b138b61a1604cd18fe31b0f4ffb28d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:31.101843 22616 tablet_bootstrap.cc:654] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:31.102790 22616 tablet_bootstrap.cc:492] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba: No bootstrap required, opened a new log
I20240502 07:58:31.102849 22616 ts_tablet_manager.cc:1367] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:31.103482 22619 tablet_bootstrap.cc:492] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2: Bootstrap starting.
I20240502 07:58:31.103577 22616 raft_consensus.cc:348] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.103688 22616 raft_consensus.cc:374] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:31.103724 22616 raft_consensus.cc:729] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Initialized, Role: FOLLOWER
I20240502 07:58:31.103885 22616 consensus_queue.cc:260] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.104238 22616 ts_tablet_manager.cc:1398] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:31.104441 22616 tablet_bootstrap.cc:492] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba: Bootstrap starting.
I20240502 07:58:31.105304 22616 tablet_bootstrap.cc:654] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:31.105307 22619 tablet_bootstrap.cc:654] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:31.105855 22615 tablet_bootstrap.cc:492] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a: No bootstrap required, opened a new log
I20240502 07:58:31.105965 22615 ts_tablet_manager.cc:1367] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240502 07:58:31.106364 22615 raft_consensus.cc:348] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.106467 22615 raft_consensus.cc:374] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:31.106503 22615 raft_consensus.cc:729] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Initialized, Role: FOLLOWER
I20240502 07:58:31.106650 22615 consensus_queue.cc:260] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.107947 22615 ts_tablet_manager.cc:1398] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:31.108160 22615 tablet_bootstrap.cc:492] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a: Bootstrap starting.
I20240502 07:58:31.109158 22615 tablet_bootstrap.cc:654] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:31.110373 22615 tablet_bootstrap.cc:492] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a: No bootstrap required, opened a new log
I20240502 07:58:31.110456 22615 ts_tablet_manager.cc:1367] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:31.110869 22615 raft_consensus.cc:348] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.110985 22615 raft_consensus.cc:374] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:31.111073 22615 raft_consensus.cc:729] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Initialized, Role: FOLLOWER
I20240502 07:58:31.111214 22615 consensus_queue.cc:260] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.111547 22615 ts_tablet_manager.cc:1398] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:31.112116 22616 tablet_bootstrap.cc:492] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba: No bootstrap required, opened a new log
I20240502 07:58:31.112234 22616 ts_tablet_manager.cc:1367] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 07:58:31.112751 22616 raft_consensus.cc:348] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.112847 22616 raft_consensus.cc:374] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:31.112882 22616 raft_consensus.cc:729] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Initialized, Role: FOLLOWER
I20240502 07:58:31.113026 22616 consensus_queue.cc:260] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.113394 22616 ts_tablet_manager.cc:1398] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:31.113652 22616 tablet_bootstrap.cc:492] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba: Bootstrap starting.
I20240502 07:58:31.114634 22616 tablet_bootstrap.cc:654] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:31.115760 22616 tablet_bootstrap.cc:492] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba: No bootstrap required, opened a new log
I20240502 07:58:31.115870 22616 ts_tablet_manager.cc:1367] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:31.116317 22616 raft_consensus.cc:348] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:31.116475 22616 raft_consensus.cc:374] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:31.116526 22616 raft_consensus.cc:729] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Initialized, Role: FOLLOWER
I20240502 07:58:31.116701 22616 consensus_queue.cc:260] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:31.116946 22616 ts_tablet_manager.cc:1398] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:31.120575 22619 tablet_bootstrap.cc:492] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2: No bootstrap required, opened a new log
I20240502 07:58:31.120678 22619 ts_tablet_manager.cc:1367] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240502 07:58:31.121135 22619 raft_consensus.cc:348] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:31.121230 22619 raft_consensus.cc:374] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:31.121253 22619 raft_consensus.cc:729] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Initialized, Role: FOLLOWER
I20240502 07:58:31.121394 22619 consensus_queue.cc:260] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:31.121762 22619 ts_tablet_manager.cc:1398] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:31.237321 22566 raft_consensus.cc:482] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:31.237483 22566 raft_consensus.cc:504] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.238451 22566 leader_election.cc:288] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:31.242255 21934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46a300254bb45d1970dfb5db271ad14" candidate_uuid: "e218451f770e4ab0a5266e60bdb30a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b4d9bd384846b2ad74c199585018ba" is_pre_election: true
I20240502 07:58:31.242486 21934 raft_consensus.cc:2454] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e218451f770e4ab0a5266e60bdb30a6a in term 0.
I20240502 07:58:31.242862 21608 leader_election.cc:302] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e218451f770e4ab0a5266e60bdb30a6a; no voters:
I20240502 07:58:31.243155 22566 raft_consensus.cc:2781] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:31.243223 22566 raft_consensus.cc:482] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:31.243260 22566 raft_consensus.cc:3037] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:31.244194 22566 raft_consensus.cc:504] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.244515 22566 leader_election.cc:288] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 election: Requested vote from peers 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:31.244911 21934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46a300254bb45d1970dfb5db271ad14" candidate_uuid: "e218451f770e4ab0a5266e60bdb30a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b4d9bd384846b2ad74c199585018ba"
I20240502 07:58:31.245023 21934 raft_consensus.cc:3037] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:31.245931 21934 raft_consensus.cc:2454] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e218451f770e4ab0a5266e60bdb30a6a in term 1.
I20240502 07:58:31.246237 21608 leader_election.cc:302] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e218451f770e4ab0a5266e60bdb30a6a; no voters:
I20240502 07:58:31.246388 22566 raft_consensus.cc:2781] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:31.246574 22566 raft_consensus.cc:686] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Becoming Leader. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Running, Role: LEADER
I20240502 07:58:31.246879 22566 consensus_queue.cc:237] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.249154 21529 catalog_manager.cc:5484] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: term changed from 0 to 1, leader changed from <none> to e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129). New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:31.413121 22566 raft_consensus.cc:482] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:31.413266 22566 raft_consensus.cc:504] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.413638 22566 leader_election.cc:288] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:31.414197 21934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b138b61a1604cd18fe31b0f4ffb28d2" candidate_uuid: "e218451f770e4ab0a5266e60bdb30a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b4d9bd384846b2ad74c199585018ba" is_pre_election: true
I20240502 07:58:31.414333 21934 raft_consensus.cc:2454] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e218451f770e4ab0a5266e60bdb30a6a in term 0.
I20240502 07:58:31.414619 21608 leader_election.cc:302] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e218451f770e4ab0a5266e60bdb30a6a; no voters:
I20240502 07:58:31.414791 22566 raft_consensus.cc:2781] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:31.414849 22566 raft_consensus.cc:482] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:31.414876 22566 raft_consensus.cc:3037] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:31.415659 22566 raft_consensus.cc:504] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.415944 22566 leader_election.cc:288] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 election: Requested vote from peers 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:31.416358 21934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b138b61a1604cd18fe31b0f4ffb28d2" candidate_uuid: "e218451f770e4ab0a5266e60bdb30a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b4d9bd384846b2ad74c199585018ba"
I20240502 07:58:31.416467 21934 raft_consensus.cc:3037] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:31.417238 21934 raft_consensus.cc:2454] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e218451f770e4ab0a5266e60bdb30a6a in term 1.
I20240502 07:58:31.417520 21608 leader_election.cc:302] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e218451f770e4ab0a5266e60bdb30a6a; no voters:
I20240502 07:58:31.417681 22566 raft_consensus.cc:2781] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:31.417791 22566 raft_consensus.cc:686] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Becoming Leader. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Running, Role: LEADER
I20240502 07:58:31.417924 22566 consensus_queue.cc:237] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.419471 21529 catalog_manager.cc:5484] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: term changed from 0 to 1, leader changed from <none> to e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129). New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:31.503284 22552 raft_consensus.cc:482] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:31.503438 22552 raft_consensus.cc:504] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:31.503875 22552 leader_election.cc:288] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:31.507395 21934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c725ad05375344c3b52682bc91090a43" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b4d9bd384846b2ad74c199585018ba" is_pre_election: true
I20240502 07:58:31.507539 21934 raft_consensus.cc:2454] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 0.
I20240502 07:58:31.507866 21739 leader_election.cc:302] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:31.508067 22552 raft_consensus.cc:2781] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:31.508134 22552 raft_consensus.cc:482] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:31.508167 22552 raft_consensus.cc:3037] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:31.508955 22552 raft_consensus.cc:504] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:31.509222 22552 leader_election.cc:288] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 election: Requested vote from peers 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:31.509572 21934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c725ad05375344c3b52682bc91090a43" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b4d9bd384846b2ad74c199585018ba"
I20240502 07:58:31.509675 21934 raft_consensus.cc:3037] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:31.510411 21934 raft_consensus.cc:2454] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 1.
I20240502 07:58:31.510689 21739 leader_election.cc:302] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:31.510843 22552 raft_consensus.cc:2781] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:31.510922 22552 raft_consensus.cc:686] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Becoming Leader. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Running, Role: LEADER
I20240502 07:58:31.511035 22552 consensus_queue.cc:237] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:31.512601 21529 catalog_manager.cc:5484] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } }
W20240502 07:58:31.658771 22630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:31.659049 22630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:31.662242 22630 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 07:58:31.686724 22566 consensus_queue.cc:1035] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:31.755573 21934 raft_consensus.cc:1261] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:31.756064 22552 consensus_queue.cc:1035] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:31.758502 21933 raft_consensus.cc:1261] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:31.758909 22566 consensus_queue.cc:1035] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 96.7646 ms
time per row: 0.0483823 ms
I20240502 07:58:31.803535 21528 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:31.809094 21782 tablet_service.cc:1467] Processing CreateTablet for tablet fd14aec4cc3f4b8fa6e3d0bb1241a240 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f8126ed45de428eaf540e5d65bd2c86]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:31.809212 21783 tablet_service.cc:1467] Processing CreateTablet for tablet 1968b639a6f54f8eb31f90d2ad0c0748 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f8126ed45de428eaf540e5d65bd2c86]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:31.809413 21782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd14aec4cc3f4b8fa6e3d0bb1241a240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:31.809363 21652 tablet_service.cc:1467] Processing CreateTablet for tablet fd14aec4cc3f4b8fa6e3d0bb1241a240 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f8126ed45de428eaf540e5d65bd2c86]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:31.809614 21652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd14aec4cc3f4b8fa6e3d0bb1241a240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:31.809756 21913 tablet_service.cc:1467] Processing CreateTablet for tablet 1968b639a6f54f8eb31f90d2ad0c0748 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f8126ed45de428eaf540e5d65bd2c86]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:31.810011 21913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1968b639a6f54f8eb31f90d2ad0c0748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:31.811697 21783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1968b639a6f54f8eb31f90d2ad0c0748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:31.812173 22684 tablet_bootstrap.cc:492] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2: Bootstrap starting.
I20240502 07:58:31.812234 21652 tablet_service.cc:1467] Processing CreateTablet for tablet e956783425bb49d2ab78be427e61781f (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f8126ed45de428eaf540e5d65bd2c86]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:31.812456 21652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e956783425bb49d2ab78be427e61781f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:31.812971 22684 tablet_bootstrap.cc:654] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:31.813576 22683 tablet_bootstrap.cc:492] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a: Bootstrap starting.
I20240502 07:58:31.814033 21782 tablet_service.cc:1467] Processing CreateTablet for tablet e956783425bb49d2ab78be427e61781f (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f8126ed45de428eaf540e5d65bd2c86]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:31.814249 21782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e956783425bb49d2ab78be427e61781f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:31.814456 22683 tablet_bootstrap.cc:654] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:31.814909 22682 tablet_bootstrap.cc:492] T 1968b639a6f54f8eb31f90d2ad0c0748 P 21b4d9bd384846b2ad74c199585018ba: Bootstrap starting.
I20240502 07:58:31.815569 22683 tablet_bootstrap.cc:492] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a: No bootstrap required, opened a new log
I20240502 07:58:31.815651 22683 ts_tablet_manager.cc:1367] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:31.815702 22684 tablet_bootstrap.cc:492] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2: No bootstrap required, opened a new log
I20240502 07:58:31.815773 22684 ts_tablet_manager.cc:1367] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 07:58:31.815897 22682 tablet_bootstrap.cc:654] T 1968b639a6f54f8eb31f90d2ad0c0748 P 21b4d9bd384846b2ad74c199585018ba: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:31.816154 22683 raft_consensus.cc:348] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:31.816254 22683 raft_consensus.cc:374] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:31.816291 22683 raft_consensus.cc:729] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Initialized, Role: FOLLOWER
I20240502 07:58:31.816257 22684 raft_consensus.cc:348] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:31.816340 22684 raft_consensus.cc:374] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:31.816380 22684 raft_consensus.cc:729] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Initialized, Role: FOLLOWER
I20240502 07:58:31.816422 22683 consensus_queue.cc:260] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:31.816483 22684 consensus_queue.cc:260] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:31.816723 22683 ts_tablet_manager.cc:1398] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:31.816761 22684 ts_tablet_manager.cc:1398] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:31.816888 22684 tablet_bootstrap.cc:492] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2: Bootstrap starting.
I20240502 07:58:31.816895 22683 tablet_bootstrap.cc:492] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a: Bootstrap starting.
I20240502 07:58:31.817021 22682 tablet_bootstrap.cc:492] T 1968b639a6f54f8eb31f90d2ad0c0748 P 21b4d9bd384846b2ad74c199585018ba: No bootstrap required, opened a new log
I20240502 07:58:31.817093 22682 ts_tablet_manager.cc:1367] T 1968b639a6f54f8eb31f90d2ad0c0748 P 21b4d9bd384846b2ad74c199585018ba: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:58:31.817520 22682 raft_consensus.cc:348] T 1968b639a6f54f8eb31f90d2ad0c0748 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.817617 22682 raft_consensus.cc:374] T 1968b639a6f54f8eb31f90d2ad0c0748 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:31.817653 22682 raft_consensus.cc:729] T 1968b639a6f54f8eb31f90d2ad0c0748 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Initialized, Role: FOLLOWER
I20240502 07:58:31.817734 22684 tablet_bootstrap.cc:654] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:31.817734 22683 tablet_bootstrap.cc:654] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:31.817778 22682 consensus_queue.cc:260] T 1968b639a6f54f8eb31f90d2ad0c0748 P 21b4d9bd384846b2ad74c199585018ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.818049 22682 ts_tablet_manager.cc:1398] T 1968b639a6f54f8eb31f90d2ad0c0748 P 21b4d9bd384846b2ad74c199585018ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:31.819103 22683 tablet_bootstrap.cc:492] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a: No bootstrap required, opened a new log
I20240502 07:58:31.819612 22683 ts_tablet_manager.cc:1367] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:31.820118 22684 tablet_bootstrap.cc:492] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2: No bootstrap required, opened a new log
I20240502 07:58:31.820112 22683 raft_consensus.cc:348] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:31.820186 22683 raft_consensus.cc:374] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:31.820210 22684 ts_tablet_manager.cc:1367] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:58:31.820219 22683 raft_consensus.cc:729] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Initialized, Role: FOLLOWER
I20240502 07:58:31.820361 22683 consensus_queue.cc:260] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:31.820647 22683 ts_tablet_manager.cc:1398] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:31.820667 22684 raft_consensus.cc:348] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.820760 22684 raft_consensus.cc:374] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:31.820792 22684 raft_consensus.cc:729] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Initialized, Role: FOLLOWER
I20240502 07:58:31.820909 22684 consensus_queue.cc:260] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:31.821184 22684 ts_tablet_manager.cc:1398] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:31.821357 22684 tablet_bootstrap.cc:492] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2: Bootstrap starting.
I20240502 07:58:31.822233 22684 tablet_bootstrap.cc:654] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:31.823474 22684 tablet_bootstrap.cc:492] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2: No bootstrap required, opened a new log
I20240502 07:58:31.823551 22684 ts_tablet_manager.cc:1367] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:58:31.823966 22684 raft_consensus.cc:348] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:31.824054 22684 raft_consensus.cc:374] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:31.824087 22684 raft_consensus.cc:729] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Initialized, Role: FOLLOWER
I20240502 07:58:31.824213 22684 consensus_queue.cc:260] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:31.824420 22684 ts_tablet_manager.cc:1398] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:31.997350 22560 raft_consensus.cc:482] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:31.997495 22560 raft_consensus.cc:504] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:31.997877 22560 leader_election.cc:288] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:31.998411 21671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e956783425bb49d2ab78be427e61781f" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a" is_pre_election: true
I20240502 07:58:31.998569 21671 raft_consensus.cc:2454] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 0.
I20240502 07:58:31.998894 21737 leader_election.cc:302] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e218451f770e4ab0a5266e60bdb30a6a, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:31.999107 22560 raft_consensus.cc:2781] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:31.999171 22560 raft_consensus.cc:482] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:31.999200 22560 raft_consensus.cc:3037] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:32.000108 22560 raft_consensus.cc:504] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.000469 22560 leader_election.cc:288] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 election: Requested vote from peers e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:32.000874 21671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e956783425bb49d2ab78be427e61781f" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a"
I20240502 07:58:32.001044 21671 raft_consensus.cc:3037] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:32.001863 21671 raft_consensus.cc:2454] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 1.
I20240502 07:58:32.002190 21737 leader_election.cc:302] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e218451f770e4ab0a5266e60bdb30a6a, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:32.002413 22560 raft_consensus.cc:2781] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:32.002518 22560 raft_consensus.cc:686] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Becoming Leader. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Running, Role: LEADER
I20240502 07:58:32.002666 22560 consensus_queue.cc:237] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.004480 21527 catalog_manager.cc:5484] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:32.085103 22552 raft_consensus.cc:482] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:32.085245 22552 raft_consensus.cc:504] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:32.085609 22552 leader_election.cc:288] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:32.086114 21933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1968b639a6f54f8eb31f90d2ad0c0748" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b4d9bd384846b2ad74c199585018ba" is_pre_election: true
I20240502 07:58:32.086253 21933 raft_consensus.cc:2454] T 1968b639a6f54f8eb31f90d2ad0c0748 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 0.
I20240502 07:58:32.086606 21739 leader_election.cc:302] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:32.086815 22552 raft_consensus.cc:2781] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:32.086881 22552 raft_consensus.cc:482] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:32.086934 22552 raft_consensus.cc:3037] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:32.087775 22552 raft_consensus.cc:504] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:32.088145 22552 leader_election.cc:288] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 election: Requested vote from peers 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:32.088536 21933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1968b639a6f54f8eb31f90d2ad0c0748" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b4d9bd384846b2ad74c199585018ba"
I20240502 07:58:32.088706 21933 raft_consensus.cc:3037] T 1968b639a6f54f8eb31f90d2ad0c0748 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:32.089543 21933 raft_consensus.cc:2454] T 1968b639a6f54f8eb31f90d2ad0c0748 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 1.
I20240502 07:58:32.089799 21739 leader_election.cc:302] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:32.089967 22552 raft_consensus.cc:2781] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:32.090073 22552 raft_consensus.cc:686] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Becoming Leader. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Running, Role: LEADER
I20240502 07:58:32.090219 22552 consensus_queue.cc:237] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:32.091866 21527 catalog_manager.cc:5484] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:32.219563 22676 raft_consensus.cc:482] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:32.219695 22676 raft_consensus.cc:504] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:32.220180 22676 leader_election.cc:288] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:32.223435 21803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd14aec4cc3f4b8fa6e3d0bb1241a240" candidate_uuid: "e218451f770e4ab0a5266e60bdb30a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8406d5a62054258be7fff420a4a4fe2" is_pre_election: true
I20240502 07:58:32.223582 21803 raft_consensus.cc:2454] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e218451f770e4ab0a5266e60bdb30a6a in term 0.
I20240502 07:58:32.223896 21606 leader_election.cc:302] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e218451f770e4ab0a5266e60bdb30a6a, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:32.224066 22676 raft_consensus.cc:2781] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:32.224129 22676 raft_consensus.cc:482] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:32.224165 22676 raft_consensus.cc:3037] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:32.224974 22676 raft_consensus.cc:504] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:32.225224 22676 leader_election.cc:288] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 election: Requested vote from peers e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:32.225597 21803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd14aec4cc3f4b8fa6e3d0bb1241a240" candidate_uuid: "e218451f770e4ab0a5266e60bdb30a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
I20240502 07:58:32.225716 21803 raft_consensus.cc:3037] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:32.226418 21803 raft_consensus.cc:2454] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e218451f770e4ab0a5266e60bdb30a6a in term 1.
I20240502 07:58:32.226668 21606 leader_election.cc:302] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e218451f770e4ab0a5266e60bdb30a6a, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:32.226816 22676 raft_consensus.cc:2781] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:32.226895 22676 raft_consensus.cc:686] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Becoming Leader. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Running, Role: LEADER
I20240502 07:58:32.227049 22676 consensus_queue.cc:237] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:32.228643 21527 catalog_manager.cc:5484] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: term changed from 0 to 1, leader changed from <none> to e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129). New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:32.378155 22695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:32.378424 22695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:32.381543 22695 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 07:58:32.403239 22552 consensus_queue.cc:1035] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:32.455766 21933 raft_consensus.cc:1261] T 1968b639a6f54f8eb31f90d2ad0c0748 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:32.456282 22560 consensus_queue.cc:1035] T 1968b639a6f54f8eb31f90d2ad0c0748 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:32.459595 22728 mvcc.cc:205] Tried to move back new op lower bound from 7023151974212874240 to 7023151972722429952. Current Snapshot: MvccSnapshot[applied={T|T < 7023151974212874240}]
I20240502 07:58:32.461247 21803 raft_consensus.cc:1261] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:32.461813 22676 consensus_queue.cc:1035] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 81.0687 ms
time per row: 0.0405343 ms
I20240502 07:58:32.506168 21527 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:32.511607 21913 tablet_service.cc:1467] Processing CreateTablet for tablet 829291de88d84d6b8ead3a450430acba (DEFAULT_TABLE table=rebalance_test_table_3 [id=aba27c7bd3a84e94b7e627dc4e8d5c6a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:32.511708 21782 tablet_service.cc:1467] Processing CreateTablet for tablet 10c7c018e7994c71a77f6d547e0c6bb5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=aba27c7bd3a84e94b7e627dc4e8d5c6a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:32.511898 21652 tablet_service.cc:1467] Processing CreateTablet for tablet 829291de88d84d6b8ead3a450430acba (DEFAULT_TABLE table=rebalance_test_table_3 [id=aba27c7bd3a84e94b7e627dc4e8d5c6a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:32.511991 21913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829291de88d84d6b8ead3a450430acba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:32.512022 21782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c7c018e7994c71a77f6d547e0c6bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:32.512166 21652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829291de88d84d6b8ead3a450430acba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:32.513753 22743 tablet_bootstrap.cc:492] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2: Bootstrap starting.
I20240502 07:58:32.514330 22744 tablet_bootstrap.cc:492] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a: Bootstrap starting.
I20240502 07:58:32.514640 22743 tablet_bootstrap.cc:654] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:32.515026 21912 tablet_service.cc:1467] Processing CreateTablet for tablet e0c289e06aba4976ab7249d72f26c0e0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=aba27c7bd3a84e94b7e627dc4e8d5c6a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:32.515269 21912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c289e06aba4976ab7249d72f26c0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:32.515271 22744 tablet_bootstrap.cc:654] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:32.515398 22745 tablet_bootstrap.cc:492] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba: Bootstrap starting.
I20240502 07:58:32.515441 21652 tablet_service.cc:1467] Processing CreateTablet for tablet e0c289e06aba4976ab7249d72f26c0e0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=aba27c7bd3a84e94b7e627dc4e8d5c6a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:32.515661 21652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c289e06aba4976ab7249d72f26c0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:32.516177 22745 tablet_bootstrap.cc:654] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:32.517107 22744 tablet_bootstrap.cc:492] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a: No bootstrap required, opened a new log
I20240502 07:58:32.517148 21913 tablet_service.cc:1467] Processing CreateTablet for tablet 10c7c018e7994c71a77f6d547e0c6bb5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=aba27c7bd3a84e94b7e627dc4e8d5c6a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:32.517197 22744 ts_tablet_manager.cc:1367] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:32.517360 21913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c7c018e7994c71a77f6d547e0c6bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:32.517683 22744 raft_consensus.cc:348] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.517776 22744 raft_consensus.cc:374] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:32.517803 22744 raft_consensus.cc:729] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Initialized, Role: FOLLOWER
I20240502 07:58:32.517931 22744 consensus_queue.cc:260] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.518201 22744 ts_tablet_manager.cc:1398] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:32.518369 22744 tablet_bootstrap.cc:492] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a: Bootstrap starting.
I20240502 07:58:32.519222 22744 tablet_bootstrap.cc:654] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:32.519832 22743 tablet_bootstrap.cc:492] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2: No bootstrap required, opened a new log
I20240502 07:58:32.519933 22743 ts_tablet_manager.cc:1367] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:58:32.520015 22745 tablet_bootstrap.cc:492] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba: No bootstrap required, opened a new log
I20240502 07:58:32.520115 22745 ts_tablet_manager.cc:1367] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 07:58:32.520444 22745 raft_consensus.cc:348] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.520530 22745 raft_consensus.cc:374] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:32.520447 22743 raft_consensus.cc:348] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:32.520551 22745 raft_consensus.cc:729] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Initialized, Role: FOLLOWER
I20240502 07:58:32.520561 22743 raft_consensus.cc:374] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:32.520591 22743 raft_consensus.cc:729] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Initialized, Role: FOLLOWER
I20240502 07:58:32.520663 22745 consensus_queue.cc:260] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.520721 22743 consensus_queue.cc:260] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:32.520949 22745 ts_tablet_manager.cc:1398] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:32.521004 22743 ts_tablet_manager.cc:1398] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:32.521121 22745 tablet_bootstrap.cc:492] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba: Bootstrap starting.
I20240502 07:58:32.521628 22744 tablet_bootstrap.cc:492] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a: No bootstrap required, opened a new log
I20240502 07:58:32.521713 22744 ts_tablet_manager.cc:1367] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:58:32.522158 22745 tablet_bootstrap.cc:654] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:32.522119 22744 raft_consensus.cc:348] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.522213 22744 raft_consensus.cc:374] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:32.522254 22744 raft_consensus.cc:729] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Initialized, Role: FOLLOWER
I20240502 07:58:32.522387 22744 consensus_queue.cc:260] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.522758 22744 ts_tablet_manager.cc:1398] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:32.523847 22745 tablet_bootstrap.cc:492] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba: No bootstrap required, opened a new log
I20240502 07:58:32.524003 22745 ts_tablet_manager.cc:1367] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 07:58:32.524667 22745 raft_consensus.cc:348] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.524762 22745 raft_consensus.cc:374] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:32.524796 22745 raft_consensus.cc:729] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Initialized, Role: FOLLOWER
I20240502 07:58:32.524940 22745 consensus_queue.cc:260] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.525225 22745 ts_tablet_manager.cc:1398] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:32.525395 22745 tablet_bootstrap.cc:492] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba: Bootstrap starting.
I20240502 07:58:32.526199 22745 tablet_bootstrap.cc:654] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:32.527407 22745 tablet_bootstrap.cc:492] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba: No bootstrap required, opened a new log
I20240502 07:58:32.527609 22745 ts_tablet_manager.cc:1367] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:32.528075 22745 raft_consensus.cc:348] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:32.528208 22745 raft_consensus.cc:374] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:32.528270 22745 raft_consensus.cc:729] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Initialized, Role: FOLLOWER
I20240502 07:58:32.528394 22745 consensus_queue.cc:260] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:32.528651 22745 ts_tablet_manager.cc:1398] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:32.528808 22541 raft_consensus.cc:482] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:32.528898 22541 raft_consensus.cc:504] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.529197 22541 leader_election.cc:288] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:32.529590 21671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829291de88d84d6b8ead3a450430acba" candidate_uuid: "21b4d9bd384846b2ad74c199585018ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a" is_pre_election: true
I20240502 07:58:32.529732 21671 raft_consensus.cc:2454] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21b4d9bd384846b2ad74c199585018ba in term 0.
I20240502 07:58:32.530016 21868 leader_election.cc:302] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e218451f770e4ab0a5266e60bdb30a6a; no voters:
I20240502 07:58:32.530295 22541 raft_consensus.cc:2781] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:32.530351 22541 raft_consensus.cc:482] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:32.530395 22541 raft_consensus.cc:3037] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:32.531234 22541 raft_consensus.cc:504] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.531479 22541 leader_election.cc:288] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 election: Requested vote from peers e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:32.531808 21671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829291de88d84d6b8ead3a450430acba" candidate_uuid: "21b4d9bd384846b2ad74c199585018ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a"
I20240502 07:58:32.531926 21671 raft_consensus.cc:3037] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:32.532649 21671 raft_consensus.cc:2454] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21b4d9bd384846b2ad74c199585018ba in term 1.
I20240502 07:58:32.532928 21868 leader_election.cc:302] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e218451f770e4ab0a5266e60bdb30a6a; no voters:
I20240502 07:58:32.533120 22541 raft_consensus.cc:2781] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:32.533339 22541 raft_consensus.cc:686] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Becoming Leader. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Running, Role: LEADER
I20240502 07:58:32.533510 22541 consensus_queue.cc:237] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.534885 21527 catalog_manager.cc:5484] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 21b4d9bd384846b2ad74c199585018ba (127.18.13.131). New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:32.658385 22676 raft_consensus.cc:482] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:32.658545 22676 raft_consensus.cc:504] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.658921 22676 leader_election.cc:288] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:32.659550 21933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c289e06aba4976ab7249d72f26c0e0" candidate_uuid: "e218451f770e4ab0a5266e60bdb30a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b4d9bd384846b2ad74c199585018ba" is_pre_election: true
I20240502 07:58:32.659713 21933 raft_consensus.cc:2454] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e218451f770e4ab0a5266e60bdb30a6a in term 0.
I20240502 07:58:32.660099 21608 leader_election.cc:302] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e218451f770e4ab0a5266e60bdb30a6a; no voters:
I20240502 07:58:32.660300 22676 raft_consensus.cc:2781] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:32.660360 22676 raft_consensus.cc:482] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:32.660418 22676 raft_consensus.cc:3037] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:32.661242 22676 raft_consensus.cc:504] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.661525 22676 leader_election.cc:288] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 election: Requested vote from peers 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:32.661952 21933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c289e06aba4976ab7249d72f26c0e0" candidate_uuid: "e218451f770e4ab0a5266e60bdb30a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b4d9bd384846b2ad74c199585018ba"
I20240502 07:58:32.662081 21933 raft_consensus.cc:3037] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:32.662835 21933 raft_consensus.cc:2454] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e218451f770e4ab0a5266e60bdb30a6a in term 1.
I20240502 07:58:32.663125 21608 leader_election.cc:302] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e218451f770e4ab0a5266e60bdb30a6a; no voters:
I20240502 07:58:32.663290 22676 raft_consensus.cc:2781] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:32.663385 22676 raft_consensus.cc:686] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Becoming Leader. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Running, Role: LEADER
I20240502 07:58:32.663529 22676 consensus_queue.cc:237] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:32.665208 21527 catalog_manager.cc:5484] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: term changed from 0 to 1, leader changed from <none> to e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129). New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:32.811573 22541 raft_consensus.cc:482] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:32.811707 22541 raft_consensus.cc:504] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:32.812090 22541 leader_election.cc:288] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:32.812530 21803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c7c018e7994c71a77f6d547e0c6bb5" candidate_uuid: "21b4d9bd384846b2ad74c199585018ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8406d5a62054258be7fff420a4a4fe2" is_pre_election: true
I20240502 07:58:32.812691 21803 raft_consensus.cc:2454] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21b4d9bd384846b2ad74c199585018ba in term 0.
I20240502 07:58:32.812999 21868 leader_election.cc:302] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:32.813190 22541 raft_consensus.cc:2781] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:32.813246 22541 raft_consensus.cc:482] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:32.813309 22541 raft_consensus.cc:3037] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:32.814107 22541 raft_consensus.cc:504] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:32.814364 22541 leader_election.cc:288] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 election: Requested vote from peers e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:32.814741 21803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c7c018e7994c71a77f6d547e0c6bb5" candidate_uuid: "21b4d9bd384846b2ad74c199585018ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
I20240502 07:58:32.814846 21803 raft_consensus.cc:3037] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:32.815559 21803 raft_consensus.cc:2454] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21b4d9bd384846b2ad74c199585018ba in term 1.
I20240502 07:58:32.815866 21868 leader_election.cc:302] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:32.816010 22541 raft_consensus.cc:2781] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:32.816102 22541 raft_consensus.cc:686] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Becoming Leader. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Running, Role: LEADER
I20240502 07:58:32.816253 22541 consensus_queue.cc:237] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } }
I20240502 07:58:32.817832 21527 catalog_manager.cc:5484] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 21b4d9bd384846b2ad74c199585018ba (127.18.13.131). New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: HEALTHY } } }
W20240502 07:58:32.986459 22756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:32.986814 22756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 07:58:32.989560 22541 consensus_queue.cc:1035] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 07:58:32.990624 22756 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 07:58:33.053916 22676 consensus_queue.cc:1035] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:33.099221 21803 raft_consensus.cc:1261] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:33.099849 22752 consensus_queue.cc:1035] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 110.016 ms
time per row: 0.0550078 ms
I20240502 07:58:33.145648 21529 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55614:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:33.151111 21782 tablet_service.cc:1467] Processing CreateTablet for tablet 56f307e7a3f34ed18e4a6f5272034279 (DEFAULT_TABLE table=rebalance_test_table_4 [id=83a6a35d6fa24f97bab7a6a64be5fe51]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:33.151234 21913 tablet_service.cc:1467] Processing CreateTablet for tablet d59002c98d3f4ab8bbea19caaf0c8419 (DEFAULT_TABLE table=rebalance_test_table_4 [id=83a6a35d6fa24f97bab7a6a64be5fe51]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:33.151454 21782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f307e7a3f34ed18e4a6f5272034279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:33.151489 21913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59002c98d3f4ab8bbea19caaf0c8419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:33.152124 21652 tablet_service.cc:1467] Processing CreateTablet for tablet 56f307e7a3f34ed18e4a6f5272034279 (DEFAULT_TABLE table=rebalance_test_table_4 [id=83a6a35d6fa24f97bab7a6a64be5fe51]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:33.152429 21652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f307e7a3f34ed18e4a6f5272034279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:33.153728 22804 tablet_bootstrap.cc:492] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba: Bootstrap starting.
I20240502 07:58:33.154376 21782 tablet_service.cc:1467] Processing CreateTablet for tablet d59002c98d3f4ab8bbea19caaf0c8419 (DEFAULT_TABLE table=rebalance_test_table_4 [id=83a6a35d6fa24f97bab7a6a64be5fe51]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:33.154670 22805 tablet_bootstrap.cc:492] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a: Bootstrap starting.
I20240502 07:58:33.154646 21651 tablet_service.cc:1467] Processing CreateTablet for tablet efe71ca9e2604316b7b93385d4a9eace (DEFAULT_TABLE table=rebalance_test_table_4 [id=83a6a35d6fa24f97bab7a6a64be5fe51]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:33.154745 22804 tablet_bootstrap.cc:654] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:33.154917 21651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe71ca9e2604316b7b93385d4a9eace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:33.154376 21783 tablet_service.cc:1467] Processing CreateTablet for tablet efe71ca9e2604316b7b93385d4a9eace (DEFAULT_TABLE table=rebalance_test_table_4 [id=83a6a35d6fa24f97bab7a6a64be5fe51]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:33.155143 21783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe71ca9e2604316b7b93385d4a9eace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:33.155506 22803 tablet_bootstrap.cc:492] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2: Bootstrap starting.
I20240502 07:58:33.155527 22805 tablet_bootstrap.cc:654] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:33.156515 22803 tablet_bootstrap.cc:654] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:33.156975 21782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59002c98d3f4ab8bbea19caaf0c8419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:33.157642 22803 tablet_bootstrap.cc:492] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2: No bootstrap required, opened a new log
I20240502 07:58:33.157737 22803 ts_tablet_manager.cc:1367] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:33.157812 22805 tablet_bootstrap.cc:492] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a: No bootstrap required, opened a new log
I20240502 07:58:33.157912 22805 ts_tablet_manager.cc:1367] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:58:33.158214 22803 raft_consensus.cc:348] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:33.158339 22803 raft_consensus.cc:374] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:33.158399 22803 raft_consensus.cc:729] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Initialized, Role: FOLLOWER
I20240502 07:58:33.158349 22805 raft_consensus.cc:348] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:33.158478 22805 raft_consensus.cc:374] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:33.158524 22805 raft_consensus.cc:729] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Initialized, Role: FOLLOWER
I20240502 07:58:33.158571 22803 consensus_queue.cc:260] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:33.158674 22805 consensus_queue.cc:260] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:33.158888 22803 ts_tablet_manager.cc:1398] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:33.158958 22805 ts_tablet_manager.cc:1398] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:33.159085 22803 tablet_bootstrap.cc:492] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2: Bootstrap starting.
I20240502 07:58:33.159163 22805 tablet_bootstrap.cc:492] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a: Bootstrap starting.
I20240502 07:58:33.160063 22805 tablet_bootstrap.cc:654] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:33.161103 22803 tablet_bootstrap.cc:654] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:33.161151 22804 tablet_bootstrap.cc:492] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba: No bootstrap required, opened a new log
I20240502 07:58:33.161247 22804 ts_tablet_manager.cc:1367] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 07:58:33.161804 22804 raft_consensus.cc:348] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:33.161952 22804 raft_consensus.cc:374] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:33.162003 22804 raft_consensus.cc:729] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Initialized, Role: FOLLOWER
I20240502 07:58:33.162034 22805 tablet_bootstrap.cc:492] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a: No bootstrap required, opened a new log
I20240502 07:58:33.162120 22805 ts_tablet_manager.cc:1367] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:33.162146 22804 consensus_queue.cc:260] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:33.162474 22804 ts_tablet_manager.cc:1398] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:33.163049 22803 tablet_bootstrap.cc:492] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2: No bootstrap required, opened a new log
I20240502 07:58:33.163141 22803 ts_tablet_manager.cc:1367] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 07:58:33.162557 22805 raft_consensus.cc:348] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:33.163223 22805 raft_consensus.cc:374] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:33.163264 22805 raft_consensus.cc:729] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Initialized, Role: FOLLOWER
I20240502 07:58:33.163380 22805 consensus_queue.cc:260] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:33.163648 22805 ts_tablet_manager.cc:1398] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:33.163643 22803 raft_consensus.cc:348] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:33.163764 22803 raft_consensus.cc:374] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:33.163854 22803 raft_consensus.cc:729] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Initialized, Role: FOLLOWER
I20240502 07:58:33.163980 22803 consensus_queue.cc:260] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:33.164315 22803 ts_tablet_manager.cc:1398] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:33.164505 22803 tablet_bootstrap.cc:492] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2: Bootstrap starting.
I20240502 07:58:33.165429 22803 tablet_bootstrap.cc:654] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:33.166604 22803 tablet_bootstrap.cc:492] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2: No bootstrap required, opened a new log
I20240502 07:58:33.166687 22803 ts_tablet_manager.cc:1367] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 07:58:33.167034 22803 raft_consensus.cc:348] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:33.167120 22803 raft_consensus.cc:374] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:33.167155 22803 raft_consensus.cc:729] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Initialized, Role: FOLLOWER
I20240502 07:58:33.167290 22803 consensus_queue.cc:260] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:33.167534 22803 ts_tablet_manager.cc:1398] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:33.357856 22677 raft_consensus.cc:482] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:33.358001 22677 raft_consensus.cc:504] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:33.358350 22677 leader_election.cc:288] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:33.358933 21803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f307e7a3f34ed18e4a6f5272034279" candidate_uuid: "e218451f770e4ab0a5266e60bdb30a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8406d5a62054258be7fff420a4a4fe2" is_pre_election: true
I20240502 07:58:33.359083 21803 raft_consensus.cc:2454] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e218451f770e4ab0a5266e60bdb30a6a in term 0.
I20240502 07:58:33.359432 21606 leader_election.cc:302] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e218451f770e4ab0a5266e60bdb30a6a, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:33.359611 22677 raft_consensus.cc:2781] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:33.359673 22677 raft_consensus.cc:482] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:33.359699 22677 raft_consensus.cc:3037] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:33.360538 22677 raft_consensus.cc:504] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:33.360802 22677 leader_election.cc:288] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 election: Requested vote from peers e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:33.361246 21803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f307e7a3f34ed18e4a6f5272034279" candidate_uuid: "e218451f770e4ab0a5266e60bdb30a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
I20240502 07:58:33.361364 21803 raft_consensus.cc:3037] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:33.362126 21803 raft_consensus.cc:2454] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e218451f770e4ab0a5266e60bdb30a6a in term 1.
I20240502 07:58:33.362327 22552 raft_consensus.cc:482] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:33.362447 21606 leader_election.cc:302] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e218451f770e4ab0a5266e60bdb30a6a, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:33.362434 22552 raft_consensus.cc:504] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:33.362591 22677 raft_consensus.cc:2781] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:33.362686 22677 raft_consensus.cc:686] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Becoming Leader. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Running, Role: LEADER
I20240502 07:58:33.362730 22552 leader_election.cc:288] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:33.362807 22677 consensus_queue.cc:237] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:33.363247 21933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59002c98d3f4ab8bbea19caaf0c8419" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b4d9bd384846b2ad74c199585018ba" is_pre_election: true
I20240502 07:58:33.363387 21933 raft_consensus.cc:2454] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 0.
I20240502 07:58:33.363682 21739 leader_election.cc:302] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:33.363906 22552 raft_consensus.cc:2781] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:33.363981 22552 raft_consensus.cc:482] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:33.364024 22552 raft_consensus.cc:3037] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:33.364745 22552 raft_consensus.cc:504] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:33.364712 21529 catalog_manager.cc:5484] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: term changed from 0 to 1, leader changed from <none> to e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129). New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:33.364996 22552 leader_election.cc:288] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 election: Requested vote from peers 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:33.365386 21933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59002c98d3f4ab8bbea19caaf0c8419" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b4d9bd384846b2ad74c199585018ba"
I20240502 07:58:33.365489 21933 raft_consensus.cc:3037] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:33.366215 21933 raft_consensus.cc:2454] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 1.
I20240502 07:58:33.366509 21739 leader_election.cc:302] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21b4d9bd384846b2ad74c199585018ba, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:33.366986 22552 raft_consensus.cc:2781] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:33.367173 22552 raft_consensus.cc:686] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Becoming Leader. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Running, Role: LEADER
I20240502 07:58:33.367316 22552 consensus_queue.cc:237] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } }
I20240502 07:58:33.368878 21529 catalog_manager.cc:5484] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:33.494511 22552 raft_consensus.cc:482] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:33.494666 22552 raft_consensus.cc:504] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:33.494978 22552 leader_election.cc:288] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:33.495411 21671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe71ca9e2604316b7b93385d4a9eace" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a" is_pre_election: true
I20240502 07:58:33.495554 21671 raft_consensus.cc:2454] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 0.
I20240502 07:58:33.495918 21737 leader_election.cc:302] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e218451f770e4ab0a5266e60bdb30a6a, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:33.496120 22552 raft_consensus.cc:2781] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:33.496187 22552 raft_consensus.cc:482] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:33.496232 22552 raft_consensus.cc:3037] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:33.497278 22552 raft_consensus.cc:504] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:33.497530 22552 leader_election.cc:288] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 election: Requested vote from peers e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:33.497893 21671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe71ca9e2604316b7b93385d4a9eace" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a"
I20240502 07:58:33.498059 21671 raft_consensus.cc:3037] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:33.498816 21671 raft_consensus.cc:2454] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 1.
I20240502 07:58:33.499078 21737 leader_election.cc:302] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e218451f770e4ab0a5266e60bdb30a6a, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:33.499240 22552 raft_consensus.cc:2781] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:33.499336 22552 raft_consensus.cc:686] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Becoming Leader. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Running, Role: LEADER
I20240502 07:58:33.499568 22552 consensus_queue.cc:237] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } }
I20240502 07:58:33.501152 21529 catalog_manager.cc:5484] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:33.636339 22816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:33.636617 22816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:33.640049 22816 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 07:58:33.726070 21803 raft_consensus.cc:1261] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:33.726702 22677 consensus_queue.cc:1035] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:33.728341 21671 raft_consensus.cc:1261] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:33.728961 22552 consensus_queue.cc:1035] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:33.734949 21933 raft_consensus.cc:1261] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:33.735599 22552 consensus_queue.cc:1035] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 93.4393 ms
time per row: 0.0467197 ms
I20240502 07:58:33.776242 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.132:40025
--local_ip_for_outbound_sockets=127.18.13.132
--tserver_master_addrs=127.18.13.190:33713
--webserver_port=44579
--webserver_interface=127.18.13.132
--builtin_ntp_servers=127.18.13.148:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 07:58:33.891963 22865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:33.892220 22865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:33.892267 22865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:33.895429 22865 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:33.895503 22865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:33.895574 22865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.132
I20240502 07:58:33.899507 22865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.132:40025
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.18.13.132
--webserver_port=44579
--tserver_master_addrs=127.18.13.190:33713
--never_fsync=true
--heap_profile_path=/tmp/kudu.22865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.132
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:33.900523 22865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:33.901690 22865 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:33.911181 22873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:33.912495 22870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:33.913578 22871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:33.914624 22865 server_base.cc:1034] running on GCE node
I20240502 07:58:33.915019 22865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:33.915593 22865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:33.916747 22865 hybrid_clock.cc:648] HybridClock initialized: now 1714636713916733 us; error 28 us; skew 500 ppm
I20240502 07:58:33.918589 22865 webserver.cc:460] Webserver started at http://127.18.13.132:44579/ using document root <none> and password file <none>
I20240502 07:58:33.919138 22865 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:33.919215 22865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:33.922549 22865 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:33.924585 22879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:33.925587 22865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:33.925705 22865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ec7796d40c8a463caa442aa67ba3d367"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:33.926198 22865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:33.944758 22865 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:33.945436 22865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:33.945605 22865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:33.946193 22865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:33.947155 22865 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:33.947203 22865 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:33.947244 22865 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:33.947270 22865 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:33.956115 22865 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.132:40025
I20240502 07:58:33.956141 22992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.132:40025 every 8 connection(s)
I20240502 07:58:33.957224 22865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:58:33.962368 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 22865
I20240502 07:58:33.962754 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.133:38189
--local_ip_for_outbound_sockets=127.18.13.133
--tserver_master_addrs=127.18.13.190:33713
--webserver_port=36873
--webserver_interface=127.18.13.133
--builtin_ntp_servers=127.18.13.148:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:33.966185 22993 heartbeater.cc:347] Connected to a master server at 127.18.13.190:33713
I20240502 07:58:33.966442 22993 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:33.966980 22993 heartbeater.cc:511] Master 127.18.13.190:33713 requested a full tablet report, sending...
I20240502 07:58:33.968278 21522 ts_manager.cc:194] Re-registered known tserver with Master: ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132:40025)
I20240502 07:58:33.969170 21522 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.132:39593
W20240502 07:58:34.073534 22996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:34.073819 22996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:34.073871 22996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:34.077035 22996 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:34.077108 22996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:34.077178 22996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.133
I20240502 07:58:34.080935 22996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.133:38189
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.18.13.133
--webserver_port=36873
--tserver_master_addrs=127.18.13.190:33713
--never_fsync=true
--heap_profile_path=/tmp/kudu.22996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.133
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:34.081871 22996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:34.083000 22996 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:34.092653 23001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:34.093519 23002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:34.094439 23004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:34.095822 22996 server_base.cc:1034] running on GCE node
I20240502 07:58:34.096261 22996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:34.096884 22996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:34.098043 22996 hybrid_clock.cc:648] HybridClock initialized: now 1714636714098022 us; error 36 us; skew 500 ppm
I20240502 07:58:34.100096 22996 webserver.cc:460] Webserver started at http://127.18.13.133:36873/ using document root <none> and password file <none>
I20240502 07:58:34.100669 22996 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:34.100749 22996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:34.104295 22996 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:58:34.106609 23010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:34.107717 22996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:58:34.107858 22996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f6079903ad93465c8741bf16583fe8c8"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:34.108390 22996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:34.121783 22996 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:34.122437 22996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:34.122596 22996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:34.123135 22996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:34.124082 22996 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:34.124145 22996 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:34.124212 22996 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:34.124243 22996 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:34.133316 22996 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.133:38189
I20240502 07:58:34.133428 23123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.133:38189 every 8 connection(s)
I20240502 07:58:34.134528 22996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:58:34.139166 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 22996
I20240502 07:58:34.139711 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.134:42113
--local_ip_for_outbound_sockets=127.18.13.134
--tserver_master_addrs=127.18.13.190:33713
--webserver_port=44811
--webserver_interface=127.18.13.134
--builtin_ntp_servers=127.18.13.148:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:34.145413 23124 heartbeater.cc:347] Connected to a master server at 127.18.13.190:33713
I20240502 07:58:34.145735 23124 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:34.146522 23124 heartbeater.cc:511] Master 127.18.13.190:33713 requested a full tablet report, sending...
I20240502 07:58:34.147877 21522 ts_manager.cc:194] Re-registered known tserver with Master: f6079903ad93465c8741bf16583fe8c8 (127.18.13.133:38189)
I20240502 07:58:34.148466 21522 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.133:39243
W20240502 07:58:34.253022 23127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:34.253257 23127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:34.253302 23127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:34.256578 23127 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:34.256688 23127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:34.256783 23127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.134
I20240502 07:58:34.260648 23127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.134:42113
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.18.13.134
--webserver_port=44811
--tserver_master_addrs=127.18.13.190:33713
--never_fsync=true
--heap_profile_path=/tmp/kudu.23127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.134
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:34.261561 23127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:34.262805 23127 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:34.272032 23135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:34.273216 23133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:34.274283 23132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:34.275274 23127 server_base.cc:1034] running on GCE node
I20240502 07:58:34.275672 23127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:34.276304 23127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:34.277457 23127 hybrid_clock.cc:648] HybridClock initialized: now 1714636714277433 us; error 46 us; skew 500 ppm
I20240502 07:58:34.279410 23127 webserver.cc:460] Webserver started at http://127.18.13.134:44811/ using document root <none> and password file <none>
I20240502 07:58:34.280038 23127 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:34.280119 23127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:34.283722 23127 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:58:34.285897 23141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:34.287092 23127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:34.287209 23127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6"
format_stamp: "Formatted at 2024-05-02 07:58:25 on dist-test-slave-f46z"
I20240502 07:58:34.287667 23127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:34.307195 23127 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:34.307965 23127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:34.308141 23127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:34.308699 23127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:34.309620 23127 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:34.309671 23127 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:34.309734 23127 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:34.309762 23127 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:34.318269 23127 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.134:42113
I20240502 07:58:34.318297 23254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.134:42113 every 8 connection(s)
I20240502 07:58:34.319383 23127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 07:58:34.326930 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 23127
I20240502 07:58:34.327378 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.135:43407
--local_ip_for_outbound_sockets=127.18.13.135
--tserver_master_addrs=127.18.13.190:33713
--webserver_port=36617
--webserver_interface=127.18.13.135
--builtin_ntp_servers=127.18.13.148:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:34.328724 23255 heartbeater.cc:347] Connected to a master server at 127.18.13.190:33713
I20240502 07:58:34.328958 23255 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:34.329506 23255 heartbeater.cc:511] Master 127.18.13.190:33713 requested a full tablet report, sending...
I20240502 07:58:34.330663 21522 ts_manager.cc:194] Re-registered known tserver with Master: c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134:42113)
I20240502 07:58:34.331550 21522 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.134:54205
W20240502 07:58:34.439898 23258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:34.440136 23258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:34.440182 23258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:34.443241 23258 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:34.443307 23258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:34.443375 23258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.135
I20240502 07:58:34.447253 23258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.135:43407
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.18.13.135
--webserver_port=36617
--tserver_master_addrs=127.18.13.190:33713
--never_fsync=true
--heap_profile_path=/tmp/kudu.23258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.135
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:34.448230 23258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:34.449340 23258 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:34.458333 23263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:34.459357 23266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:34.460583 23264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:34.461745 23258 server_base.cc:1034] running on GCE node
I20240502 07:58:34.462167 23258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:34.462774 23258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:34.463924 23258 hybrid_clock.cc:648] HybridClock initialized: now 1714636714463906 us; error 33 us; skew 500 ppm
I20240502 07:58:34.465862 23258 webserver.cc:460] Webserver started at http://127.18.13.135:36617/ using document root <none> and password file <none>
I20240502 07:58:34.466439 23258 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:34.466516 23258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:34.470216 23258 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:58:34.472242 23272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:34.473315 23258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:34.473416 23258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f97f8632bbe64638bb1ad2679849f434"
format_stamp: "Formatted at 2024-05-02 07:58:26 on dist-test-slave-f46z"
I20240502 07:58:34.473881 23258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:34.486925 23258 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:34.487618 23258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:34.487819 23258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:34.488391 23258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:34.489413 23258 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:34.489461 23258 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:34.489509 23258 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:34.489548 23258 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:34.498301 23258 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.135:43407
I20240502 07:58:34.498346 23385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.135:43407 every 8 connection(s)
I20240502 07:58:34.499415 23258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 07:58:34.503929 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 23258
I20240502 07:58:34.504124 18486 test_util.cc:268] Using random seed: -129404997
I20240502 07:58:34.509196 23386 heartbeater.cc:347] Connected to a master server at 127.18.13.190:33713
I20240502 07:58:34.509456 23386 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:34.510053 23386 heartbeater.cc:511] Master 127.18.13.190:33713 requested a full tablet report, sending...
I20240502 07:58:34.510962 21522 ts_manager.cc:194] Re-registered known tserver with Master: f97f8632bbe64638bb1ad2679849f434 (127.18.13.135:43407)
I20240502 07:58:34.511656 21522 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.135:33051
I20240502 07:58:34.514026 18486 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 07:58:34.514242 18486 test_util.cc:268] Using random seed: -129394880
I20240502 07:58:34.523507 18486 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 07:58:34.523682 18486 test_util.cc:268] Using random seed: -129385439
I20240502 07:58:34.533670 18486 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 07:58:34.533838 18486 test_util.cc:268] Using random seed: -129375283
I20240502 07:58:34.551378 18486 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 07:58:34.551544 18486 test_util.cc:268] Using random seed: -129357577
I20240502 07:58:34.567193 18486 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240502 07:58:34.939548 21651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:34.962083 22927 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:34.965166 23319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:34.971324 22993 heartbeater.cc:503] Master 127.18.13.190:33713 was elected leader, sending a full tablet report...
I20240502 07:58:34.983232 23189 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:35.005947 21913 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:35.050720 21782 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:58:35.051499 23058 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:35.150966 23124 heartbeater.cc:503] Master 127.18.13.190:33713 was elected leader, sending a full tablet report...
I20240502 07:58:35.312043 21651 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:35.315840 21913 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:35.316802 21782 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:58:35.332954 23319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:35.333671 23255 heartbeater.cc:503] Master 127.18.13.190:33713 was elected leader, sending a full tablet report...
I20240502 07:58:35.336374 23189 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:35.345755 23058 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:35.358569 22927 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:35.438784 21672 consensus_queue.cc:237] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } }
I20240502 07:58:35.443512 21801 raft_consensus.cc:1261] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240502 07:58:35.445324 23468 consensus_queue.cc:1035] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240502 07:58:35.451850 23475 raft_consensus.cc:2932] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } } }
I20240502 07:58:35.454391 21801 raft_consensus.cc:2932] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } } }
W20240502 07:58:35.456280 21605 consensus_peers.cc:485] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a -> Peer f97f8632bbe64638bb1ad2679849f434 (127.18.13.135:43407): Couldn't send request to peer f97f8632bbe64638bb1ad2679849f434. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56f307e7a3f34ed18e4a6f5272034279. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:35.456751 21529 catalog_manager.cc:5484] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: config changed from index -1 to 29, NON_VOTER f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:35.474558 21668 consensus_queue.cc:237] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } }
I20240502 07:58:35.477331 21933 raft_consensus.cc:1261] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240502 07:58:35.480580 23427 consensus_queue.cc:1035] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240502 07:58:35.484079 23468 raft_consensus.cc:2932] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index -1 to 31, NON_VOTER f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } } }
I20240502 07:58:35.486239 21933 raft_consensus.cc:2932] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index -1 to 31, NON_VOTER f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } } }
W20240502 07:58:35.488322 21607 consensus_peers.cc:485] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a -> Peer f6079903ad93465c8741bf16583fe8c8 (127.18.13.133:38189): Couldn't send request to peer f6079903ad93465c8741bf16583fe8c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0c289e06aba4976ab7249d72f26c0e0. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:35.498437 21523 catalog_manager.cc:5484] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: config changed from index -1 to 31, NON_VOTER f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:35.513808 23386 heartbeater.cc:503] Master 127.18.13.190:33713 was elected leader, sending a full tablet report...
I20240502 07:58:35.547701 21672 consensus_queue.cc:237] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } }
I20240502 07:58:35.549832 21801 raft_consensus.cc:1261] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20240502 07:58:35.551820 23475 consensus_queue.cc:1035] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 33, Time since last communication: 0.000s
W20240502 07:58:35.556565 21605 consensus_peers.cc:485] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a -> Peer ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132:40025): Couldn't send request to peer ec7796d40c8a463caa442aa67ba3d367. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd14aec4cc3f4b8fa6e3d0bb1241a240. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:35.559216 23468 raft_consensus.cc:2932] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index -1 to 35, NON_VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } } }
I20240502 07:58:35.565160 21523 catalog_manager.cc:5484] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: config changed from index -1 to 35, NON_VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:35.568747 21803 raft_consensus.cc:2932] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index -1 to 35, NON_VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } } }
I20240502 07:58:35.583062 21801 consensus_queue.cc:237] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } }
I20240502 07:58:35.589773 21671 raft_consensus.cc:1261] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20240502 07:58:35.591032 23413 consensus_queue.cc:1035] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 37, Time since last communication: 0.000s
W20240502 07:58:35.595341 21737 consensus_peers.cc:485] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 -> Peer c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134:42113): Couldn't send request to peer c0ee0b72cadc4ebaa2e32a046a107ce6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0029538492ea4f25a822f18de9c83c54. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:35.598253 23436 raft_consensus.cc:2932] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index -1 to 39, NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } } }
I20240502 07:58:35.602579 21523 catalog_manager.cc:5484] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index -1 to 39, NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:35.609988 21670 raft_consensus.cc:2932] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index -1 to 39, NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) added. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } } }
I20240502 07:58:35.625861 21672 consensus_queue.cc:237] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } }
W20240502 07:58:35.631510 21605 consensus_peers.cc:485] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a -> Peer f97f8632bbe64638bb1ad2679849f434 (127.18.13.135:43407): Couldn't send request to peer f97f8632bbe64638bb1ad2679849f434. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b138b61a1604cd18fe31b0f4ffb28d2. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:35.632457 21933 raft_consensus.cc:1261] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240502 07:58:35.633728 23427 consensus_queue.cc:1035] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240502 07:58:35.640200 23468 raft_consensus.cc:2932] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } } }
I20240502 07:58:35.643147 21523 catalog_manager.cc:5484] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: config changed from index -1 to 26, NON_VOTER f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:35.643297 21932 raft_consensus.cc:2932] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } } }
I20240502 07:58:35.710865 21933 consensus_queue.cc:237] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } }
I20240502 07:58:35.713604 21668 raft_consensus.cc:1261] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20240502 07:58:35.714861 23484 consensus_queue.cc:1035] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
W20240502 07:58:35.718932 21869 consensus_peers.cc:485] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba -> Peer f6079903ad93465c8741bf16583fe8c8 (127.18.13.133:38189): Couldn't send request to peer f6079903ad93465c8741bf16583fe8c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3f58339fd2c4528a2decb910ea73e21. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:35.719436 23449 raft_consensus.cc:2932] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index -1 to 35, NON_VOTER f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } } }
I20240502 07:58:35.722308 21671 raft_consensus.cc:2932] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index -1 to 35, NON_VOTER f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } } }
I20240502 07:58:35.722805 21523 catalog_manager.cc:5484] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba reported cstate change: config changed from index -1 to 35, NON_VOTER f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:35.742573 21801 consensus_queue.cc:237] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } }
I20240502 07:58:35.747655 21671 raft_consensus.cc:1261] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
I20240502 07:58:35.749115 23459 consensus_queue.cc:1035] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
W20240502 07:58:35.750437 21736 consensus_peers.cc:485] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 -> Peer ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132:40025): Couldn't send request to peer ec7796d40c8a463caa442aa67ba3d367. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ed04eb8c46f4cf4bf721d75baa8d534. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:35.760105 22856 raft_consensus.cc:2932] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } } }
I20240502 07:58:35.763718 21529 catalog_manager.cc:5484] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index -1 to 41, NON_VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:35.769632 21668 raft_consensus.cc:2932] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } } }
I20240502 07:58:35.814785 21803 consensus_queue.cc:237] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.43, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } }
I20240502 07:58:35.816520 21668 raft_consensus.cc:1261] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 1 index: 44. (index mismatch)
W20240502 07:58:35.816879 21737 consensus_peers.cc:485] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 -> Peer c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134:42113): Couldn't send request to peer c0ee0b72cadc4ebaa2e32a046a107ce6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efe71ca9e2604316b7b93385d4a9eace. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:35.817337 23520 consensus_queue.cc:1035] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20240502 07:58:35.821738 23520 raft_consensus.cc:2932] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } } }
I20240502 07:58:35.822769 21668 raft_consensus.cc:2932] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } } }
I20240502 07:58:35.825290 21529 catalog_manager.cc:5484] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index -1 to 44, NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:35.856882 21803 consensus_queue.cc:237] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } }
W20240502 07:58:35.858808 21736 consensus_peers.cc:485] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 -> Peer ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132:40025): Couldn't send request to peer ec7796d40c8a463caa442aa67ba3d367. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c725ad05375344c3b52682bc91090a43. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:35.861867 21931 raft_consensus.cc:1261] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
I20240502 07:58:35.862612 23413 consensus_queue.cc:1035] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 39, Time since last communication: 0.000s
I20240502 07:58:35.864843 23436 raft_consensus.cc:2932] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } } }
I20240502 07:58:35.868275 21523 catalog_manager.cc:5484] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index -1 to 41, NON_VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:35.868439 21933 raft_consensus.cc:2932] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index -1 to 41, NON_VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } } }
I20240502 07:58:35.918349 21933 consensus_queue.cc:237] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } }
I20240502 07:58:35.924389 21671 raft_consensus.cc:1261] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240502 07:58:35.926087 22752 consensus_queue.cc:1035] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
W20240502 07:58:35.927110 21867 consensus_peers.cc:485] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba -> Peer f97f8632bbe64638bb1ad2679849f434 (127.18.13.135:43407): Couldn't send request to peer f97f8632bbe64638bb1ad2679849f434. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 829291de88d84d6b8ead3a450430acba. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:35.929100 23449 raft_consensus.cc:2932] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index -1 to 64, NON_VOTER f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } } }
I20240502 07:58:35.931159 21671 raft_consensus.cc:2932] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index -1 to 64, NON_VOTER f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) added. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } } }
I20240502 07:58:35.935119 21523 catalog_manager.cc:5484] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba reported cstate change: config changed from index -1 to 64, NON_VOTER f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:35.948781 23628 ts_tablet_manager.cc:902] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8: Initiating tablet copy from peer e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:35.953119 23628 tablet_copy_client.cc:308] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:40667
I20240502 07:58:35.970180 23629 ts_tablet_manager.cc:902] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367: Initiating tablet copy from peer e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:35.976524 23629 tablet_copy_client.cc:308] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:40667
I20240502 07:58:35.990303 21692 tablet_copy_service.cc:140] P e218451f770e4ab0a5266e60bdb30a6a: Received BeginTabletCopySession request for tablet fd14aec4cc3f4b8fa6e3d0bb1241a240 from peer ec7796d40c8a463caa442aa67ba3d367 ({username='slave'} at 127.18.13.132:39687)
I20240502 07:58:35.990474 21692 tablet_copy_service.cc:161] P e218451f770e4ab0a5266e60bdb30a6a: Beginning new tablet copy session on tablet fd14aec4cc3f4b8fa6e3d0bb1241a240 from peer ec7796d40c8a463caa442aa67ba3d367 at {username='slave'} at 127.18.13.132:39687: session id = ec7796d40c8a463caa442aa67ba3d367-fd14aec4cc3f4b8fa6e3d0bb1241a240
I20240502 07:58:35.992026 21692 tablet_copy_source_session.cc:205] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:36.001586 23629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd14aec4cc3f4b8fa6e3d0bb1241a240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:36.019762 21668 consensus_queue.cc:237] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.43, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } }
I20240502 07:58:36.021157 21934 raft_consensus.cc:1261] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 1 index: 44. (index mismatch)
I20240502 07:58:36.021553 23468 consensus_queue.cc:1035] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20240502 07:58:36.025590 23642 ts_tablet_manager.cc:902] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434: Initiating tablet copy from peer e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:36.027724 22677 raft_consensus.cc:2932] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } } }
I20240502 07:58:36.030478 23642 tablet_copy_client.cc:308] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:40667
I20240502 07:58:36.029865 23629 tablet_copy_client.cc:783] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:36.031002 23629 tablet_copy_client.cc:647] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:36.034080 21529 catalog_manager.cc:5484] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: config changed from index -1 to 44, NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:36.037117 21692 tablet_copy_service.cc:140] P e218451f770e4ab0a5266e60bdb30a6a: Received BeginTabletCopySession request for tablet e0c289e06aba4976ab7249d72f26c0e0 from peer f6079903ad93465c8741bf16583fe8c8 ({username='slave'} at 127.18.13.133:34883)
I20240502 07:58:36.037227 21692 tablet_copy_service.cc:161] P e218451f770e4ab0a5266e60bdb30a6a: Beginning new tablet copy session on tablet e0c289e06aba4976ab7249d72f26c0e0 from peer f6079903ad93465c8741bf16583fe8c8 at {username='slave'} at 127.18.13.133:34883: session id = f6079903ad93465c8741bf16583fe8c8-e0c289e06aba4976ab7249d72f26c0e0
I20240502 07:58:36.039880 21934 raft_consensus.cc:2932] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Committing config change with OpId 1.44: config changed from index -1 to 44, NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } } }
I20240502 07:58:36.041944 21692 tablet_copy_source_session.cc:205] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:36.033255 23629 tablet_copy_client.cc:523] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 07:58:36.048197 21606 consensus_peers.cc:485] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a -> Peer c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134:42113): Couldn't send request to peer c0ee0b72cadc4ebaa2e32a046a107ce6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e46a300254bb45d1970dfb5db271ad14. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:36.048928 23628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c289e06aba4976ab7249d72f26c0e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:36.053836 23629 tablet_bootstrap.cc:492] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367: Bootstrap starting.
I20240502 07:58:36.068590 21692 tablet_copy_service.cc:140] P e218451f770e4ab0a5266e60bdb30a6a: Received BeginTabletCopySession request for tablet 56f307e7a3f34ed18e4a6f5272034279 from peer f97f8632bbe64638bb1ad2679849f434 ({username='slave'} at 127.18.13.135:47475)
I20240502 07:58:36.068718 21692 tablet_copy_service.cc:161] P e218451f770e4ab0a5266e60bdb30a6a: Beginning new tablet copy session on tablet 56f307e7a3f34ed18e4a6f5272034279 from peer f97f8632bbe64638bb1ad2679849f434 at {username='slave'} at 127.18.13.135:47475: session id = f97f8632bbe64638bb1ad2679849f434-56f307e7a3f34ed18e4a6f5272034279
I20240502 07:58:36.070147 21692 tablet_copy_source_session.cc:205] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:36.074739 23642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f307e7a3f34ed18e4a6f5272034279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:36.078447 21934 consensus_queue.cc:237] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } }
I20240502 07:58:36.079949 21801 raft_consensus.cc:1261] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:36.080374 23617 consensus_queue.cc:1035] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 07:58:36.084328 23628 tablet_copy_client.cc:783] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:36.086426 23628 tablet_copy_client.cc:647] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8: tablet copy: Starting download of 1 WAL segments...
W20240502 07:58:36.087538 21867 consensus_peers.cc:485] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba -> Peer ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132:40025): Couldn't send request to peer ec7796d40c8a463caa442aa67ba3d367. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01e44d65c0314f0a9d30a2587723c290. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:36.088738 23617 raft_consensus.cc:2932] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } } }
I20240502 07:58:36.089164 21801 raft_consensus.cc:2932] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } } }
I20240502 07:58:36.091958 21529 catalog_manager.cc:5484] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index -1 to 2, NON_VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } } }
I20240502 07:58:36.093230 23655 ts_tablet_manager.cc:902] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6: Initiating tablet copy from peer e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:36.093582 23656 ts_tablet_manager.cc:902] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434: Initiating tablet copy from peer e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:36.094192 23656 tablet_copy_client.cc:308] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:40667
I20240502 07:58:36.095414 21692 tablet_copy_service.cc:140] P e218451f770e4ab0a5266e60bdb30a6a: Received BeginTabletCopySession request for tablet 4b138b61a1604cd18fe31b0f4ffb28d2 from peer f97f8632bbe64638bb1ad2679849f434 ({username='slave'} at 127.18.13.135:47475)
I20240502 07:58:36.095536 21692 tablet_copy_service.cc:161] P e218451f770e4ab0a5266e60bdb30a6a: Beginning new tablet copy session on tablet 4b138b61a1604cd18fe31b0f4ffb28d2 from peer f97f8632bbe64638bb1ad2679849f434 at {username='slave'} at 127.18.13.135:47475: session id = f97f8632bbe64638bb1ad2679849f434-4b138b61a1604cd18fe31b0f4ffb28d2
I20240502 07:58:36.096372 23655 tablet_copy_client.cc:308] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:40891
I20240502 07:58:36.096484 21692 tablet_copy_source_session.cc:205] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:36.097532 23628 tablet_copy_client.cc:523] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:36.100422 23628 tablet_bootstrap.cc:492] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8: Bootstrap starting.
I20240502 07:58:36.113075 21823 tablet_copy_service.cc:140] P e8406d5a62054258be7fff420a4a4fe2: Received BeginTabletCopySession request for tablet 0029538492ea4f25a822f18de9c83c54 from peer c0ee0b72cadc4ebaa2e32a046a107ce6 ({username='slave'} at 127.18.13.134:39315)
I20240502 07:58:36.113242 21823 tablet_copy_service.cc:161] P e8406d5a62054258be7fff420a4a4fe2: Beginning new tablet copy session on tablet 0029538492ea4f25a822f18de9c83c54 from peer c0ee0b72cadc4ebaa2e32a046a107ce6 at {username='slave'} at 127.18.13.134:39315: session id = c0ee0b72cadc4ebaa2e32a046a107ce6-0029538492ea4f25a822f18de9c83c54
I20240502 07:58:36.114754 21823 tablet_copy_source_session.cc:205] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:36.117451 23642 tablet_copy_client.cc:783] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:36.117725 23642 tablet_copy_client.cc:647] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:36.125772 23628 log.cc:824] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:36.129786 23629 log.cc:824] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:36.140381 23656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b138b61a1604cd18fe31b0f4ffb28d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:36.147172 23642 tablet_copy_client.cc:523] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:36.160825 23655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0029538492ea4f25a822f18de9c83c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:36.164110 23642 tablet_bootstrap.cc:492] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434: Bootstrap starting.
I20240502 07:58:36.166759 23656 tablet_copy_client.cc:783] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:36.166991 23656 tablet_copy_client.cc:647] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:36.169152 23656 tablet_copy_client.cc:523] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:36.193629 23656 tablet_bootstrap.cc:492] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434: Bootstrap starting.
I20240502 07:58:36.224594 23655 tablet_copy_client.cc:783] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:36.224907 23655 tablet_copy_client.cc:647] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:36.227247 23655 tablet_copy_client.cc:523] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:36.228161 23656 log.cc:824] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:36.231495 23655 tablet_bootstrap.cc:492] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6: Bootstrap starting.
I20240502 07:58:36.242348 23674 ts_tablet_manager.cc:902] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8: Initiating tablet copy from peer 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:36.243573 23674 tablet_copy_client.cc:308] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:38811
I20240502 07:58:36.249379 21954 tablet_copy_service.cc:140] P 21b4d9bd384846b2ad74c199585018ba: Received BeginTabletCopySession request for tablet f3f58339fd2c4528a2decb910ea73e21 from peer f6079903ad93465c8741bf16583fe8c8 ({username='slave'} at 127.18.13.133:45027)
I20240502 07:58:36.249543 21954 tablet_copy_service.cc:161] P 21b4d9bd384846b2ad74c199585018ba: Beginning new tablet copy session on tablet f3f58339fd2c4528a2decb910ea73e21 from peer f6079903ad93465c8741bf16583fe8c8 at {username='slave'} at 127.18.13.133:45027: session id = f6079903ad93465c8741bf16583fe8c8-f3f58339fd2c4528a2decb910ea73e21
I20240502 07:58:36.251060 21954 tablet_copy_source_session.cc:205] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:36.253837 23674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3f58339fd2c4528a2decb910ea73e21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:36.259763 23674 tablet_copy_client.cc:783] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:36.260000 23674 tablet_copy_client.cc:647] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:36.270361 23655 log.cc:824] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:36.277843 23674 tablet_copy_client.cc:523] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:36.280454 23674 tablet_bootstrap.cc:492] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8: Bootstrap starting.
I20240502 07:58:36.286127 23629 tablet_bootstrap.cc:492] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367: Bootstrap replayed 1/1 log segments. Stats: ops{read=45 overwritten=0 applied=45 ignored=0} inserts{seen=730 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:36.287688 23682 ts_tablet_manager.cc:902] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6: Initiating tablet copy from peer e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:36.288444 23682 tablet_copy_client.cc:308] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:40891
I20240502 07:58:36.289489 21823 tablet_copy_service.cc:140] P e8406d5a62054258be7fff420a4a4fe2: Received BeginTabletCopySession request for tablet efe71ca9e2604316b7b93385d4a9eace from peer c0ee0b72cadc4ebaa2e32a046a107ce6 ({username='slave'} at 127.18.13.134:39315)
I20240502 07:58:36.289628 21823 tablet_copy_service.cc:161] P e8406d5a62054258be7fff420a4a4fe2: Beginning new tablet copy session on tablet efe71ca9e2604316b7b93385d4a9eace from peer c0ee0b72cadc4ebaa2e32a046a107ce6 at {username='slave'} at 127.18.13.134:39315: session id = c0ee0b72cadc4ebaa2e32a046a107ce6-efe71ca9e2604316b7b93385d4a9eace
I20240502 07:58:36.290678 21823 tablet_copy_source_session.cc:205] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:36.293082 23682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe71ca9e2604316b7b93385d4a9eace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:36.308333 23629 tablet_bootstrap.cc:492] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367: Bootstrap complete.
I20240502 07:58:36.308851 23629 ts_tablet_manager.cc:1367] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367: Time spent bootstrapping tablet: real 0.260s user 0.074s sys 0.000s
I20240502 07:58:36.313864 23682 tablet_copy_client.cc:783] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:36.314208 23682 tablet_copy_client.cc:647] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:36.317250 23682 tablet_copy_client.cc:523] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:36.318838 23682 tablet_bootstrap.cc:492] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6: Bootstrap starting.
I20240502 07:58:36.311702 23629 raft_consensus.cc:348] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } }
I20240502 07:58:36.319360 23629 raft_consensus.cc:729] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ec7796d40c8a463caa442aa67ba3d367, State: Initialized, Role: LEARNER
I20240502 07:58:36.324291 23629 consensus_queue.cc:260] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 1.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } }
I20240502 07:58:36.325635 23629 ts_tablet_manager.cc:1398] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20240502 07:58:36.332948 21692 tablet_copy_service.cc:342] P e218451f770e4ab0a5266e60bdb30a6a: Request end of tablet copy session ec7796d40c8a463caa442aa67ba3d367-fd14aec4cc3f4b8fa6e3d0bb1241a240 received from {username='slave'} at 127.18.13.132:39687
I20240502 07:58:36.333087 21692 tablet_copy_service.cc:433] P e218451f770e4ab0a5266e60bdb30a6a: ending tablet copy session ec7796d40c8a463caa442aa67ba3d367-fd14aec4cc3f4b8fa6e3d0bb1241a240 on tablet fd14aec4cc3f4b8fa6e3d0bb1241a240 with peer ec7796d40c8a463caa442aa67ba3d367
I20240502 07:58:36.341125 23629 ts_tablet_manager.cc:902] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367: Initiating tablet copy from peer e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:36.341686 23629 tablet_copy_client.cc:308] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:40891
I20240502 07:58:36.373579 23697 ts_tablet_manager.cc:902] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367: Initiating tablet copy from peer e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:36.375995 23695 ts_tablet_manager.cc:902] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434: Initiating tablet copy from peer 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:36.377432 23695 tablet_copy_client.cc:308] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:38811
I20240502 07:58:36.379626 21801 consensus_queue.cc:237] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 1.51, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } }
W20240502 07:58:36.381415 21736 consensus_peers.cc:485] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 -> Peer ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132:40025): Couldn't send request to peer ec7796d40c8a463caa442aa67ba3d367. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d59002c98d3f4ab8bbea19caaf0c8419. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:36.381922 23697 tablet_copy_client.cc:308] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:40891
I20240502 07:58:36.383474 21954 tablet_copy_service.cc:140] P 21b4d9bd384846b2ad74c199585018ba: Received BeginTabletCopySession request for tablet 829291de88d84d6b8ead3a450430acba from peer f97f8632bbe64638bb1ad2679849f434 ({username='slave'} at 127.18.13.135:36213)
I20240502 07:58:36.383596 21954 tablet_copy_service.cc:161] P 21b4d9bd384846b2ad74c199585018ba: Beginning new tablet copy session on tablet 829291de88d84d6b8ead3a450430acba from peer f97f8632bbe64638bb1ad2679849f434 at {username='slave'} at 127.18.13.135:36213: session id = f97f8632bbe64638bb1ad2679849f434-829291de88d84d6b8ead3a450430acba
I20240502 07:58:36.384357 23628 tablet_bootstrap.cc:492] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=60 overwritten=0 applied=60 ignored=0} inserts{seen=732 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:36.384721 21822 tablet_copy_service.cc:140] P e8406d5a62054258be7fff420a4a4fe2: Received BeginTabletCopySession request for tablet c725ad05375344c3b52682bc91090a43 from peer ec7796d40c8a463caa442aa67ba3d367 ({username='slave'} at 127.18.13.132:37655)
I20240502 07:58:36.384773 23628 tablet_bootstrap.cc:492] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8: Bootstrap complete.
I20240502 07:58:36.384814 21822 tablet_copy_service.cc:161] P e8406d5a62054258be7fff420a4a4fe2: Beginning new tablet copy session on tablet c725ad05375344c3b52682bc91090a43 from peer ec7796d40c8a463caa442aa67ba3d367 at {username='slave'} at 127.18.13.132:37655: session id = ec7796d40c8a463caa442aa67ba3d367-c725ad05375344c3b52682bc91090a43
I20240502 07:58:36.384918 21954 tablet_copy_source_session.cc:205] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:36.385138 23628 ts_tablet_manager.cc:1367] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8: Time spent bootstrapping tablet: real 0.285s user 0.079s sys 0.005s
I20240502 07:58:36.385232 21823 tablet_copy_service.cc:140] P e8406d5a62054258be7fff420a4a4fe2: Received BeginTabletCopySession request for tablet 2ed04eb8c46f4cf4bf721d75baa8d534 from peer ec7796d40c8a463caa442aa67ba3d367 ({username='slave'} at 127.18.13.132:37655)
I20240502 07:58:36.385305 21823 tablet_copy_service.cc:161] P e8406d5a62054258be7fff420a4a4fe2: Beginning new tablet copy session on tablet 2ed04eb8c46f4cf4bf721d75baa8d534 from peer ec7796d40c8a463caa442aa67ba3d367 at {username='slave'} at 127.18.13.132:37655: session id = ec7796d40c8a463caa442aa67ba3d367-2ed04eb8c46f4cf4bf721d75baa8d534
I20240502 07:58:36.385720 21822 tablet_copy_source_session.cc:205] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:36.386137 21823 tablet_copy_source_session.cc:205] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:36.387357 23695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829291de88d84d6b8ead3a450430acba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:36.388959 23697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c725ad05375344c3b52682bc91090a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:36.389897 23629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed04eb8c46f4cf4bf721d75baa8d534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:36.392205 23629 tablet_copy_client.cc:783] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:36.392357 23629 tablet_copy_client.cc:647] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:36.394765 21934 raft_consensus.cc:1261] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 51. Preceding OpId from leader: term: 1 index: 52. (index mismatch)
I20240502 07:58:36.395352 23672 consensus_queue.cc:1035] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20240502 07:58:36.397029 23672 raft_consensus.cc:2932] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.52: config changed from index -1 to 52, NON_VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } } }
I20240502 07:58:36.398838 23697 tablet_copy_client.cc:783] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:36.399057 23697 tablet_copy_client.cc:647] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:36.399619 21933 raft_consensus.cc:2932] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Committing config change with OpId 1.52: config changed from index -1 to 52, NON_VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) added. New config: { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } } }
I20240502 07:58:36.402505 21529 catalog_manager.cc:5484] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba reported cstate change: config changed from index -1 to 52, NON_VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } } }
I20240502 07:58:36.406051 23628 raft_consensus.cc:348] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } }
I20240502 07:58:36.407874 23628 raft_consensus.cc:729] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f6079903ad93465c8741bf16583fe8c8, State: Initialized, Role: LEARNER
I20240502 07:58:36.411837 23695 tablet_copy_client.cc:783] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:36.412153 23695 tablet_copy_client.cc:647] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:36.412539 23628 consensus_queue.cc:260] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 60, Last appended: 1.60, Last appended by leader: 60, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } }
I20240502 07:58:36.415565 23628 ts_tablet_manager.cc:1398] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20240502 07:58:36.418490 21692 tablet_copy_service.cc:342] P e218451f770e4ab0a5266e60bdb30a6a: Request end of tablet copy session f6079903ad93465c8741bf16583fe8c8-e0c289e06aba4976ab7249d72f26c0e0 received from {username='slave'} at 127.18.13.133:34883
I20240502 07:58:36.418604 21692 tablet_copy_service.cc:433] P e218451f770e4ab0a5266e60bdb30a6a: ending tablet copy session f6079903ad93465c8741bf16583fe8c8-e0c289e06aba4976ab7249d72f26c0e0 on tablet e0c289e06aba4976ab7249d72f26c0e0 with peer f6079903ad93465c8741bf16583fe8c8
I20240502 07:58:36.420568 23629 tablet_copy_client.cc:523] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:36.422188 23629 tablet_bootstrap.cc:492] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367: Bootstrap starting.
I20240502 07:58:36.425066 23697 tablet_copy_client.cc:523] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:36.426563 23697 tablet_bootstrap.cc:492] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367: Bootstrap starting.
I20240502 07:58:36.464697 22947 raft_consensus.cc:1203] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.34->[1.35-1.53] Dedup: 1.45->[1.46-1.53]
I20240502 07:58:36.468799 23695 tablet_copy_client.cc:523] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:36.470494 23695 tablet_bootstrap.cc:492] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434: Bootstrap starting.
I20240502 07:58:36.479187 23656 tablet_bootstrap.cc:492] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=710 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:36.479648 23656 tablet_bootstrap.cc:492] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434: Bootstrap complete.
I20240502 07:58:36.495249 23656 ts_tablet_manager.cc:1367] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434: Time spent bootstrapping tablet: real 0.302s user 0.059s sys 0.000s
I20240502 07:58:36.498917 23656 raft_consensus.cc:348] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } }
I20240502 07:58:36.499209 23656 raft_consensus.cc:729] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f97f8632bbe64638bb1ad2679849f434, State: Initialized, Role: LEARNER
I20240502 07:58:36.510818 23642 tablet_bootstrap.cc:492] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434: Bootstrap replayed 1/1 log segments. Stats: ops{read=44 overwritten=0 applied=43 ignored=0} inserts{seen=728 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 07:58:36.511524 23715 ts_tablet_manager.cc:902] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367: Initiating tablet copy from peer 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:36.512215 23715 tablet_copy_client.cc:308] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:38811
I20240502 07:58:36.520041 23656 consensus_queue.cc:260] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 1.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } }
I20240502 07:58:36.523905 21954 tablet_copy_service.cc:140] P 21b4d9bd384846b2ad74c199585018ba: Received BeginTabletCopySession request for tablet 01e44d65c0314f0a9d30a2587723c290 from peer ec7796d40c8a463caa442aa67ba3d367 ({username='slave'} at 127.18.13.132:47915)
I20240502 07:58:36.524037 21954 tablet_copy_service.cc:161] P 21b4d9bd384846b2ad74c199585018ba: Beginning new tablet copy session on tablet 01e44d65c0314f0a9d30a2587723c290 from peer ec7796d40c8a463caa442aa67ba3d367 at {username='slave'} at 127.18.13.132:47915: session id = ec7796d40c8a463caa442aa67ba3d367-01e44d65c0314f0a9d30a2587723c290
I20240502 07:58:36.525089 21954 tablet_copy_source_session.cc:205] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:36.526316 23715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e44d65c0314f0a9d30a2587723c290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:36.528827 23715 tablet_copy_client.cc:783] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:36.528986 23715 tablet_copy_client.cc:647] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:36.529965 23642 tablet_bootstrap.cc:492] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434: Bootstrap complete.
I20240502 07:58:36.530000 23715 tablet_copy_client.cc:523] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:36.530201 23642 ts_tablet_manager.cc:1367] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434: Time spent bootstrapping tablet: real 0.366s user 0.066s sys 0.002s
I20240502 07:58:36.530877 23642 raft_consensus.cc:348] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } }
I20240502 07:58:36.531291 23715 tablet_bootstrap.cc:492] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367: Bootstrap starting.
I20240502 07:58:36.534009 23642 raft_consensus.cc:729] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f97f8632bbe64638bb1ad2679849f434, State: Initialized, Role: LEARNER
I20240502 07:58:36.534153 23642 consensus_queue.cc:260] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 43, Last appended: 1.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } }
I20240502 07:58:36.534722 23642 ts_tablet_manager.cc:1398] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 07:58:36.536334 23078 raft_consensus.cc:1203] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.30->[1.31-1.70] Dedup: 1.60->[1.61-1.70]
I20240502 07:58:36.536736 21692 tablet_copy_service.cc:342] P e218451f770e4ab0a5266e60bdb30a6a: Request end of tablet copy session f97f8632bbe64638bb1ad2679849f434-56f307e7a3f34ed18e4a6f5272034279 received from {username='slave'} at 127.18.13.135:47475
I20240502 07:58:36.536815 21692 tablet_copy_service.cc:433] P e218451f770e4ab0a5266e60bdb30a6a: ending tablet copy session f97f8632bbe64638bb1ad2679849f434-56f307e7a3f34ed18e4a6f5272034279 on tablet 56f307e7a3f34ed18e4a6f5272034279 with peer f97f8632bbe64638bb1ad2679849f434
I20240502 07:58:36.543344 23656 ts_tablet_manager.cc:1398] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20240502 07:58:36.544240 22677 raft_consensus.cc:1050] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a: attempting to promote NON_VOTER ec7796d40c8a463caa442aa67ba3d367 to VOTER
I20240502 07:58:36.544739 22677 consensus_queue.cc:237] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.55, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } }
I20240502 07:58:36.546011 21692 tablet_copy_service.cc:342] P e218451f770e4ab0a5266e60bdb30a6a: Request end of tablet copy session f97f8632bbe64638bb1ad2679849f434-4b138b61a1604cd18fe31b0f4ffb28d2 received from {username='slave'} at 127.18.13.135:47475
I20240502 07:58:36.546097 21692 tablet_copy_service.cc:433] P e218451f770e4ab0a5266e60bdb30a6a: ending tablet copy session f97f8632bbe64638bb1ad2679849f434-4b138b61a1604cd18fe31b0f4ffb28d2 on tablet 4b138b61a1604cd18fe31b0f4ffb28d2 with peer f97f8632bbe64638bb1ad2679849f434
I20240502 07:58:36.546533 23340 raft_consensus.cc:1203] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.48] Dedup: 1.38->[1.39-1.48]
I20240502 07:58:36.549716 22946 raft_consensus.cc:1261] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240502 07:58:36.550282 23427 consensus_queue.cc:1035] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20240502 07:58:36.554806 21801 raft_consensus.cc:1261] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240502 07:58:36.555116 23427 raft_consensus.cc:2932] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.56: config changed from index 35 to 56, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:36.556324 23339 raft_consensus.cc:1203] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.47] Dedup: 1.44->[1.45-1.47]
I20240502 07:58:36.557590 22946 raft_consensus.cc:2932] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index 35 to 56, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:36.558662 23468 consensus_queue.cc:1035] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
I20240502 07:58:36.560014 21801 raft_consensus.cc:2932] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index 35 to 56, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:36.560809 21529 catalog_manager.cc:5484] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 reported cstate change: config changed from index 35 to 56, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:36.565186 21668 consensus_queue.cc:237] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 1.56, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } }
I20240502 07:58:36.566648 22946 raft_consensus.cc:1261] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 1 index: 57. (index mismatch)
I20240502 07:58:36.567088 22677 consensus_queue.cc:1035] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s
I20240502 07:58:36.573047 23469 raft_consensus.cc:2932] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.57: config changed from index 56 to 57, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:36.574862 21515 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fd14aec4cc3f4b8fa6e3d0bb1241a240 with cas_config_opid_index 56: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:36.575237 23715 tablet_bootstrap.cc:492] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:36.575502 23715 tablet_bootstrap.cc:492] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367: Bootstrap complete.
I20240502 07:58:36.575918 22946 raft_consensus.cc:2932] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Committing config change with OpId 1.57: config changed from index 56 to 57, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New config: { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:36.578609 21529 catalog_manager.cc:5484] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 reported cstate change: config changed from index 56 to 57, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 57 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:36.580293 23715 ts_tablet_manager.cc:1367] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.004s
I20240502 07:58:36.580994 23715 raft_consensus.cc:348] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } }
I20240502 07:58:36.581123 23715 raft_consensus.cc:729] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ec7796d40c8a463caa442aa67ba3d367, State: Initialized, Role: LEARNER
I20240502 07:58:36.581223 23715 consensus_queue.cc:260] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } }
I20240502 07:58:36.581802 23715 ts_tablet_manager.cc:1398] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:36.584153 21954 tablet_copy_service.cc:342] P 21b4d9bd384846b2ad74c199585018ba: Request end of tablet copy session ec7796d40c8a463caa442aa67ba3d367-01e44d65c0314f0a9d30a2587723c290 received from {username='slave'} at 127.18.13.132:47915
I20240502 07:58:36.584254 21954 tablet_copy_service.cc:433] P 21b4d9bd384846b2ad74c199585018ba: ending tablet copy session ec7796d40c8a463caa442aa67ba3d367-01e44d65c0314f0a9d30a2587723c290 on tablet 01e44d65c0314f0a9d30a2587723c290 with peer ec7796d40c8a463caa442aa67ba3d367
I20240502 07:58:36.588843 22677 raft_consensus.cc:1050] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a: attempting to promote NON_VOTER f97f8632bbe64638bb1ad2679849f434 to VOTER
I20240502 07:58:36.589282 22677 consensus_queue.cc:237] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 1.48, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } }
I20240502 07:58:36.593962 23340 raft_consensus.cc:1261] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
I20240502 07:58:36.595518 21934 raft_consensus.cc:1261] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
I20240502 07:58:36.596022 23427 consensus_queue.cc:1035] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20240502 07:58:36.596669 23427 consensus_queue.cc:1035] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20240502 07:58:36.598223 23747 ts_tablet_manager.cc:902] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6: Initiating tablet copy from peer e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:36.598814 23747 tablet_copy_client.cc:308] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:40667
I20240502 07:58:36.599474 23427 raft_consensus.cc:2932] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.49: config changed from index 26 to 49, f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.600482 21934 raft_consensus.cc:2932] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index 26 to 49, f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.603466 21526 catalog_manager.cc:5484] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: config changed from index 26 to 49, f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:36.604207 23674 tablet_bootstrap.cc:492] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=55 overwritten=0 applied=54 ignored=0} inserts{seen=685 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 07:58:36.604617 23674 tablet_bootstrap.cc:492] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8: Bootstrap complete.
I20240502 07:58:36.604809 23674 ts_tablet_manager.cc:1367] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8: Time spent bootstrapping tablet: real 0.324s user 0.062s sys 0.003s
I20240502 07:58:36.613057 23674 raft_consensus.cc:348] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } }
I20240502 07:58:36.613444 23674 raft_consensus.cc:729] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f6079903ad93465c8741bf16583fe8c8, State: Initialized, Role: LEARNER
I20240502 07:58:36.613581 23674 consensus_queue.cc:260] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 54, Last appended: 1.55, Last appended by leader: 55, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } }
I20240502 07:58:36.613837 23674 ts_tablet_manager.cc:1398] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 07:58:36.614464 21954 tablet_copy_service.cc:342] P 21b4d9bd384846b2ad74c199585018ba: Request end of tablet copy session f6079903ad93465c8741bf16583fe8c8-f3f58339fd2c4528a2decb910ea73e21 received from {username='slave'} at 127.18.13.133:45027
I20240502 07:58:36.614547 21954 tablet_copy_service.cc:433] P 21b4d9bd384846b2ad74c199585018ba: ending tablet copy session f6079903ad93465c8741bf16583fe8c8-f3f58339fd2c4528a2decb910ea73e21 on tablet f3f58339fd2c4528a2decb910ea73e21 with peer f6079903ad93465c8741bf16583fe8c8
I20240502 07:58:36.620414 21692 tablet_copy_service.cc:140] P e218451f770e4ab0a5266e60bdb30a6a: Received BeginTabletCopySession request for tablet e46a300254bb45d1970dfb5db271ad14 from peer c0ee0b72cadc4ebaa2e32a046a107ce6 ({username='slave'} at 127.18.13.134:58139)
I20240502 07:58:36.620558 21692 tablet_copy_service.cc:161] P e218451f770e4ab0a5266e60bdb30a6a: Beginning new tablet copy session on tablet e46a300254bb45d1970dfb5db271ad14 from peer c0ee0b72cadc4ebaa2e32a046a107ce6 at {username='slave'} at 127.18.13.134:58139: session id = c0ee0b72cadc4ebaa2e32a046a107ce6-e46a300254bb45d1970dfb5db271ad14
I20240502 07:58:36.621510 21692 tablet_copy_source_session.cc:205] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:36.622601 23747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46a300254bb45d1970dfb5db271ad14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:36.625545 23747 tablet_copy_client.cc:783] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:36.625846 23747 tablet_copy_client.cc:647] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:36.626645 23340 raft_consensus.cc:2932] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index 26 to 49, f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.629508 21668 consensus_queue.cc:237] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 1.52, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } }
I20240502 07:58:36.630841 23747 tablet_copy_client.cc:523] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:36.631208 23340 raft_consensus.cc:1261] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 52. Preceding OpId from leader: term: 1 index: 53. (index mismatch)
I20240502 07:58:36.631812 22677 consensus_queue.cc:1035] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 51, Time since last communication: 0.000s
I20240502 07:58:36.633942 23747 tablet_bootstrap.cc:492] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6: Bootstrap starting.
I20240502 07:58:36.634725 23475 raft_consensus.cc:2932] T 4b138b61a1604cd18fe31b0f4ffb28d2 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.53: config changed from index 49 to 53, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.634914 23340 raft_consensus.cc:2932] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Committing config change with OpId 1.53: config changed from index 49 to 53, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.636652 21515 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b138b61a1604cd18fe31b0f4ffb28d2 with cas_config_opid_index 49: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:36.637979 21526 catalog_manager.cc:5484] T 4b138b61a1604cd18fe31b0f4ffb28d2 P f97f8632bbe64638bb1ad2679849f434 reported cstate change: config changed from index 49 to 53, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.640794 21913 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b138b61a1604cd18fe31b0f4ffb28d2 with delete_type TABLET_DATA_TOMBSTONED (TS 21b4d9bd384846b2ad74c199585018ba not found in new config with opid_index 53) from {username='slave'} at 127.0.0.1:40950
I20240502 07:58:36.654687 21782 tablet_service.cc:1514] Processing DeleteTablet for tablet fd14aec4cc3f4b8fa6e3d0bb1241a240 with delete_type TABLET_DATA_TOMBSTONED (TS e8406d5a62054258be7fff420a4a4fe2 not found in new config with opid_index 57) from {username='slave'} at 127.0.0.1:51132
I20240502 07:58:36.655561 23695 tablet_bootstrap.cc:492] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434: Bootstrap replayed 1/1 log segments. Stats: ops{read=75 overwritten=0 applied=75 ignored=0} inserts{seen=760 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:36.655944 23695 tablet_bootstrap.cc:492] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434: Bootstrap complete.
I20240502 07:58:36.656169 23695 ts_tablet_manager.cc:1367] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434: Time spent bootstrapping tablet: real 0.186s user 0.077s sys 0.004s
I20240502 07:58:36.656630 23695 raft_consensus.cc:348] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } }
I20240502 07:58:36.656791 23695 raft_consensus.cc:729] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f97f8632bbe64638bb1ad2679849f434, State: Initialized, Role: LEARNER
I20240502 07:58:36.656899 23695 consensus_queue.cc:260] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 1.75, Last appended by leader: 75, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } }
I20240502 07:58:36.657194 23695 ts_tablet_manager.cc:1398] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:36.660513 21954 tablet_copy_service.cc:342] P 21b4d9bd384846b2ad74c199585018ba: Request end of tablet copy session f97f8632bbe64638bb1ad2679849f434-829291de88d84d6b8ead3a450430acba received from {username='slave'} at 127.18.13.135:36213
I20240502 07:58:36.660614 21954 tablet_copy_service.cc:433] P 21b4d9bd384846b2ad74c199585018ba: ending tablet copy session f97f8632bbe64638bb1ad2679849f434-829291de88d84d6b8ead3a450430acba on tablet 829291de88d84d6b8ead3a450430acba with peer f97f8632bbe64638bb1ad2679849f434
I20240502 07:58:36.672228 23761 tablet_replica.cc:330] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba: stopping tablet replica
I20240502 07:58:36.672696 23761 raft_consensus.cc:2229] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:36.673094 23761 pending_rounds.cc:62] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba: Trying to abort 1 pending ops.
I20240502 07:58:36.673230 23427 raft_consensus.cc:1050] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a: attempting to promote NON_VOTER f97f8632bbe64638bb1ad2679849f434 to VOTER
I20240502 07:58:36.673177 23761 pending_rounds.cc:68] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba: Aborting op as it isn't in flight: id { term: 1 index: 52 } timestamp: 7023151991297585152 op_type: WRITE_OP write_request { tablet_id: "4b138b61a1604cd18fe31b0f4ffb28d2" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000b\272\232}\024:UK\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""888278a25b7bb016" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023151991273242624 authz_token { token_data: "\010\326\221\315\261\006\"3\n\005slave\022*\n 5d61f2326983407fa0ce10f3bd2dcf48\020\001\030\001 \001(\001" signature: "<redacted>""\032\256F\0360\032\t\033:\337^K\242\240\305\364\006\317K`\331>\344\205\342\260/\270`>\3558`\2229G;\371\'[\267\343M:\033\'\330\267O\nE\2338\257\010\007wi\267\263\326\316%\036" signing_key_seq_num: 0 } } request_id { client_id: "8b70c325b8134002a9da01ca7d556abf" seq_no: 155 first_incomplete_seq_no: 155 attempt_no: 0 }
I20240502 07:58:36.673566 23761 raft_consensus.cc:2258] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:36.675583 23427 consensus_queue.cc:237] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.48, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } }
I20240502 07:58:36.676862 23427 raft_consensus.cc:1050] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a: attempting to promote NON_VOTER f6079903ad93465c8741bf16583fe8c8 to VOTER
I20240502 07:58:36.677196 23427 consensus_queue.cc:237] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } }
I20240502 07:58:36.677729 23655 tablet_bootstrap.cc:492] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6: Bootstrap replayed 1/1 log segments. Stats: ops{read=52 overwritten=0 applied=51 ignored=0} inserts{seen=723 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 07:58:36.678376 23078 raft_consensus.cc:1261] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240502 07:58:36.679078 23763 tablet_replica.cc:330] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2: stopping tablet replica
I20240502 07:58:36.679167 23468 consensus_queue.cc:1035] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 70, Time since last communication: 0.000s
I20240502 07:58:36.679646 23340 raft_consensus.cc:1261] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
I20240502 07:58:36.680135 23763 raft_consensus.cc:2229] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:36.680362 23763 raft_consensus.cc:2258] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:36.680999 23469 consensus_queue.cc:1035] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 47, Time since last communication: 0.000s
I20240502 07:58:36.681183 21803 raft_consensus.cc:1261] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
I20240502 07:58:36.681807 22677 consensus_queue.cc:1035] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20240502 07:58:36.682895 23427 raft_consensus.cc:2932] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index 31 to 72, f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:36.683416 23468 raft_consensus.cc:2932] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.49: config changed from index 29 to 49, f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.683691 23763 ts_tablet_manager.cc:1875] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:36.685976 23340 raft_consensus.cc:2932] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index 29 to 49, f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.688179 21803 raft_consensus.cc:2932] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index 29 to 49, f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.688179 21526 catalog_manager.cc:5484] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: config changed from index 31 to 72, f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:36.689886 23078 raft_consensus.cc:2932] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 31 to 72, f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:36.690810 21523 catalog_manager.cc:5484] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434 reported cstate change: config changed from index 29 to 49, f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.692189 23682 tablet_bootstrap.cc:492] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6: Bootstrap replayed 1/1 log segments. Stats: ops{read=60 overwritten=0 applied=60 ignored=0} inserts{seen=732 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:36.698251 23682 tablet_bootstrap.cc:492] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6: Bootstrap complete.
I20240502 07:58:36.698654 23682 ts_tablet_manager.cc:1367] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6: Time spent bootstrapping tablet: real 0.380s user 0.069s sys 0.004s
I20240502 07:58:36.698784 21668 consensus_queue.cc:237] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } }
I20240502 07:58:36.700774 23078 raft_consensus.cc:1261] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8 [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240502 07:58:36.701174 23469 consensus_queue.cc:1035] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240502 07:58:36.701344 23682 raft_consensus.cc:348] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } }
I20240502 07:58:36.701623 23682 raft_consensus.cc:729] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c0ee0b72cadc4ebaa2e32a046a107ce6, State: Initialized, Role: LEARNER
I20240502 07:58:36.701984 23682 consensus_queue.cc:260] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 60, Last appended: 1.60, Last appended by leader: 60, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } }
I20240502 07:58:36.702966 23682 ts_tablet_manager.cc:1398] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 07:58:36.703027 23763 ts_tablet_manager.cc:1888] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.56
I20240502 07:58:36.703202 23763 log.cc:1196] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e8406d5a62054258be7fff420a4a4fe2: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/fd14aec4cc3f4b8fa6e3d0bb1241a240
I20240502 07:58:36.702915 21668 consensus_queue.cc:237] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.50, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } }
I20240502 07:58:36.704721 21823 tablet_copy_service.cc:342] P e8406d5a62054258be7fff420a4a4fe2: Request end of tablet copy session c0ee0b72cadc4ebaa2e32a046a107ce6-efe71ca9e2604316b7b93385d4a9eace received from {username='slave'} at 127.18.13.134:39315
I20240502 07:58:36.704838 21823 tablet_copy_service.cc:433] P e8406d5a62054258be7fff420a4a4fe2: ending tablet copy session c0ee0b72cadc4ebaa2e32a046a107ce6-efe71ca9e2604316b7b93385d4a9eace on tablet efe71ca9e2604316b7b93385d4a9eace with peer c0ee0b72cadc4ebaa2e32a046a107ce6
I20240502 07:58:36.704835 23340 raft_consensus.cc:1261] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240502 07:58:36.705233 23077 raft_consensus.cc:2932] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 72 to 74, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:36.705906 23475 raft_consensus.cc:2932] T e0c289e06aba4976ab7249d72f26c0e0 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index 72 to 74, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:36.707518 21515 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e0c289e06aba4976ab7249d72f26c0e0 with cas_config_opid_index 72: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:36.707935 23427 consensus_queue.cc:1035] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20240502 07:58:36.708943 23655 tablet_bootstrap.cc:492] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6: Bootstrap complete.
I20240502 07:58:36.709161 23655 ts_tablet_manager.cc:1367] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6: Time spent bootstrapping tablet: real 0.478s user 0.076s sys 0.002s
I20240502 07:58:36.709668 23469 raft_consensus.cc:2932] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.51: config changed from index 49 to 51, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.710088 23340 raft_consensus.cc:2932] T 56f307e7a3f34ed18e4a6f5272034279 P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index 49 to 51, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.711108 21515 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56f307e7a3f34ed18e4a6f5272034279 with cas_config_opid_index 49: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:36.712815 21523 catalog_manager.cc:5484] T e0c289e06aba4976ab7249d72f26c0e0 P f6079903ad93465c8741bf16583fe8c8 reported cstate change: config changed from index 72 to 74, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:36.713670 23655 raft_consensus.cc:348] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } }
I20240502 07:58:36.715327 23655 raft_consensus.cc:729] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c0ee0b72cadc4ebaa2e32a046a107ce6, State: Initialized, Role: LEARNER
I20240502 07:58:36.715477 23655 consensus_queue.cc:260] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 51, Last appended: 1.52, Last appended by leader: 52, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } }
I20240502 07:58:36.715821 23655 ts_tablet_manager.cc:1398] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 07:58:36.717998 21515 catalog_manager.cc:4830] TS e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891): tablet fd14aec4cc3f4b8fa6e3d0bb1241a240 (table rebalance_test_table_2 [id=4f8126ed45de428eaf540e5d65bd2c86]) successfully deleted
I20240502 07:58:36.719060 21823 tablet_copy_service.cc:342] P e8406d5a62054258be7fff420a4a4fe2: Request end of tablet copy session c0ee0b72cadc4ebaa2e32a046a107ce6-0029538492ea4f25a822f18de9c83c54 received from {username='slave'} at 127.18.13.134:39315
I20240502 07:58:36.719147 21823 tablet_copy_service.cc:433] P e8406d5a62054258be7fff420a4a4fe2: ending tablet copy session c0ee0b72cadc4ebaa2e32a046a107ce6-0029538492ea4f25a822f18de9c83c54 on tablet 0029538492ea4f25a822f18de9c83c54 with peer c0ee0b72cadc4ebaa2e32a046a107ce6
I20240502 07:58:36.741453 21933 raft_consensus.cc:1261] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240502 07:58:36.745141 21529 catalog_manager.cc:5484] T 56f307e7a3f34ed18e4a6f5272034279 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: config changed from index 49 to 51, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:36.750494 21913 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c289e06aba4976ab7249d72f26c0e0 with delete_type TABLET_DATA_TOMBSTONED (TS 21b4d9bd384846b2ad74c199585018ba not found in new config with opid_index 74) from {username='slave'} at 127.0.0.1:40950
I20240502 07:58:36.751078 21782 tablet_service.cc:1514] Processing DeleteTablet for tablet 56f307e7a3f34ed18e4a6f5272034279 with delete_type TABLET_DATA_TOMBSTONED (TS e8406d5a62054258be7fff420a4a4fe2 not found in new config with opid_index 51) from {username='slave'} at 127.0.0.1:51132
I20240502 07:58:36.751446 23763 tablet_replica.cc:330] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2: stopping tablet replica
I20240502 07:58:36.751593 23763 raft_consensus.cc:2229] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:36.751724 23763 raft_consensus.cc:2258] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:36.752652 23340 raft_consensus.cc:1203] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.63->[1.64-1.81] Dedup: 1.75->[1.76-1.81]
I20240502 07:58:36.753923 23761 ts_tablet_manager.cc:1875] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:36.758857 23763 ts_tablet_manager.cc:1875] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:36.761260 23763 ts_tablet_manager.cc:1888] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.50
I20240502 07:58:36.761338 23763 log.cc:1196] T 56f307e7a3f34ed18e4a6f5272034279 P e8406d5a62054258be7fff420a4a4fe2: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/56f307e7a3f34ed18e4a6f5272034279
I20240502 07:58:36.764497 21515 catalog_manager.cc:4830] TS e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891): tablet 56f307e7a3f34ed18e4a6f5272034279 (table rebalance_test_table_4 [id=83a6a35d6fa24f97bab7a6a64be5fe51]) successfully deleted
I20240502 07:58:36.776522 23449 raft_consensus.cc:1050] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba: attempting to promote NON_VOTER f97f8632bbe64638bb1ad2679849f434 to VOTER
I20240502 07:58:36.777114 23449 consensus_queue.cc:237] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } }
I20240502 07:58:36.780409 23715 ts_tablet_manager.cc:902] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367: Initiating tablet copy from peer e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:36.780920 23715 tablet_copy_client.cc:308] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:40891
I20240502 07:58:36.781312 21669 raft_consensus.cc:1261] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240502 07:58:36.781492 23339 raft_consensus.cc:1261] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240502 07:58:36.782480 23617 consensus_queue.cc:1035] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240502 07:58:36.782805 23617 consensus_queue.cc:1035] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240502 07:58:36.784957 23484 raft_consensus.cc:2932] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index 64 to 84, f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.787457 23339 raft_consensus.cc:2932] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 64 to 84, f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.790403 21529 catalog_manager.cc:5484] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 reported cstate change: config changed from index 64 to 84, f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.794899 21934 consensus_queue.cc:237] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 1.84, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } }
I20240502 07:58:36.796167 23697 tablet_bootstrap.cc:492] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367: Bootstrap replayed 1/1 log segments. Stats: ops{read=51 overwritten=0 applied=50 ignored=0} inserts{seen=681 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 07:58:36.796662 23697 tablet_bootstrap.cc:492] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367: Bootstrap complete.
I20240502 07:58:36.796909 23697 ts_tablet_manager.cc:1367] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367: Time spent bootstrapping tablet: real 0.370s user 0.065s sys 0.000s
I20240502 07:58:36.797418 23697 raft_consensus.cc:348] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } }
I20240502 07:58:36.797624 23340 raft_consensus.cc:1261] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240502 07:58:36.797763 23697 raft_consensus.cc:729] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ec7796d40c8a463caa442aa67ba3d367, State: Initialized, Role: LEARNER
I20240502 07:58:36.797892 23697 consensus_queue.cc:260] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 50, Last appended: 1.51, Last appended by leader: 51, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } }
I20240502 07:58:36.798173 23697 ts_tablet_manager.cc:1398] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:36.798480 23484 consensus_queue.cc:1035] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20240502 07:58:36.800338 23449 raft_consensus.cc:2932] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.85: config changed from index 84 to 85, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.800844 23338 raft_consensus.cc:2932] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 84 to 85, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.802168 21517 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 829291de88d84d6b8ead3a450430acba with cas_config_opid_index 84: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:36.804066 21523 catalog_manager.cc:5484] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 reported cstate change: config changed from index 84 to 85, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:36.806475 21823 tablet_copy_service.cc:140] P e8406d5a62054258be7fff420a4a4fe2: Received BeginTabletCopySession request for tablet d59002c98d3f4ab8bbea19caaf0c8419 from peer ec7796d40c8a463caa442aa67ba3d367 ({username='slave'} at 127.18.13.132:37655)
I20240502 07:58:36.806588 21823 tablet_copy_service.cc:161] P e8406d5a62054258be7fff420a4a4fe2: Beginning new tablet copy session on tablet d59002c98d3f4ab8bbea19caaf0c8419 from peer ec7796d40c8a463caa442aa67ba3d367 at {username='slave'} at 127.18.13.132:37655: session id = ec7796d40c8a463caa442aa67ba3d367-d59002c98d3f4ab8bbea19caaf0c8419
I20240502 07:58:36.807493 21823 tablet_copy_source_session.cc:205] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:36.807940 21823 tablet_copy_service.cc:342] P e8406d5a62054258be7fff420a4a4fe2: Request end of tablet copy session ec7796d40c8a463caa442aa67ba3d367-c725ad05375344c3b52682bc91090a43 received from {username='slave'} at 127.18.13.132:37655
I20240502 07:58:36.808027 21823 tablet_copy_service.cc:433] P e8406d5a62054258be7fff420a4a4fe2: ending tablet copy session ec7796d40c8a463caa442aa67ba3d367-c725ad05375344c3b52682bc91090a43 on tablet c725ad05375344c3b52682bc91090a43 with peer ec7796d40c8a463caa442aa67ba3d367
I20240502 07:58:36.808728 23715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59002c98d3f4ab8bbea19caaf0c8419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:36.811081 23715 tablet_copy_client.cc:783] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:36.811244 23715 tablet_copy_client.cc:647] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:36.818991 21651 tablet_service.cc:1514] Processing DeleteTablet for tablet 829291de88d84d6b8ead3a450430acba with delete_type TABLET_DATA_TOMBSTONED (TS e218451f770e4ab0a5266e60bdb30a6a not found in new config with opid_index 85) from {username='slave'} at 127.0.0.1:46142
I20240502 07:58:36.826910 23077 raft_consensus.cc:1203] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.34->[1.35-1.63] Dedup: 1.55->[1.56-1.63]
I20240502 07:58:36.829998 23209 raft_consensus.cc:1203] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.43->[1.44-1.71] Dedup: 1.60->[1.61-1.71]
I20240502 07:58:36.830384 23715 tablet_copy_client.cc:523] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:36.831910 23715 tablet_bootstrap.cc:492] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367: Bootstrap starting.
I20240502 07:58:36.834300 23761 ts_tablet_manager.cc:1888] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.52
I20240502 07:58:36.834439 23761 log.cc:1196] T 4b138b61a1604cd18fe31b0f4ffb28d2 P 21b4d9bd384846b2ad74c199585018ba: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/4b138b61a1604cd18fe31b0f4ffb28d2
I20240502 07:58:36.835276 23788 tablet_replica.cc:330] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a: stopping tablet replica
I20240502 07:58:36.837723 21517 catalog_manager.cc:4830] TS 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811): tablet 4b138b61a1604cd18fe31b0f4ffb28d2 (table rebalance_test_table_1 [id=5d61f2326983407fa0ce10f3bd2dcf48]) successfully deleted
I20240502 07:58:36.838470 22946 raft_consensus.cc:1203] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.40->[1.41-1.58] Dedup: 1.51->[1.52-1.58]
I20240502 07:58:36.844957 23788 raft_consensus.cc:2229] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:36.851962 23788 pending_rounds.cc:62] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a: Trying to abort 1 pending ops.
I20240502 07:58:36.852063 23788 pending_rounds.cc:68] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a: Aborting op as it isn't in flight: id { term: 1 index: 84 } timestamp: 7023151991917981696 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "829291de88d84d6b8ead3a450430acba" old_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } } } new_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } } }
I20240502 07:58:36.852252 23788 raft_consensus.cc:2894] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Aborting config change with OpId 1.84: Aborted: Op aborted
I20240502 07:58:36.852324 23788 raft_consensus.cc:2866] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 07:58:36.852376 23788 raft_consensus.cc:2258] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:36.861056 23629 tablet_bootstrap.cc:492] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367: Bootstrap replayed 1/1 log segments. Stats: ops{read=64 overwritten=0 applied=63 ignored=0} inserts{seen=748 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 07:58:36.861136 23761 tablet_replica.cc:330] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba: stopping tablet replica
I20240502 07:58:36.861294 23761 raft_consensus.cc:2229] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:36.861395 23761 pending_rounds.cc:62] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba: Trying to abort 1 pending ops.
I20240502 07:58:36.861393 23629 tablet_bootstrap.cc:492] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367: Bootstrap complete.
I20240502 07:58:36.861426 23761 pending_rounds.cc:68] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba: Aborting op as it isn't in flight: id { term: 1 index: 71 } timestamp: 7023151991483822080 op_type: WRITE_OP write_request { tablet_id: "e0c289e06aba4976ab7249d72f26c0e0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\313\215\324\034xL\266\007\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""bea26f6db4833c1d" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023151990880862208 authz_token { token_data: "\010\326\221\315\261\006\"3\n\005slave\022*\n aba27c7bd3a84e94b7e627dc4e8d5c6a\020\001\030\001 \001(\001" signature: "<redacted>""\217T\004\251\351\3003\316\007)\233\321\361S\206\253M&\242\237\305\316.\356\r\366\365<\203L\346\3434\213\217\370\212\205\023\037^\013\302\257\213e<\360!_c+\205#\345\375\364\314\265\364\231cH\376" signing_key_seq_num: 0 } } request_id { client_id: "ca2679ae34144721a4f75c6b15c85d40" seq_no: 222 first_incomplete_seq_no: 222 attempt_no: 0 }
I20240502 07:58:36.861595 23629 ts_tablet_manager.cc:1367] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367: Time spent bootstrapping tablet: real 0.439s user 0.066s sys 0.008s
I20240502 07:58:36.861732 23761 raft_consensus.cc:2258] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:36.862056 23629 raft_consensus.cc:348] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } }
I20240502 07:58:36.867580 23629 raft_consensus.cc:729] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ec7796d40c8a463caa442aa67ba3d367, State: Initialized, Role: LEARNER
I20240502 07:58:36.867719 23629 consensus_queue.cc:260] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 63, Last appended: 1.64, Last appended by leader: 64, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } }
I20240502 07:58:36.870947 23629 ts_tablet_manager.cc:1398] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 07:58:36.874863 21823 tablet_copy_service.cc:342] P e8406d5a62054258be7fff420a4a4fe2: Request end of tablet copy session ec7796d40c8a463caa442aa67ba3d367-2ed04eb8c46f4cf4bf721d75baa8d534 received from {username='slave'} at 127.18.13.132:37655
I20240502 07:58:36.874948 21823 tablet_copy_service.cc:433] P e8406d5a62054258be7fff420a4a4fe2: ending tablet copy session ec7796d40c8a463caa442aa67ba3d367-2ed04eb8c46f4cf4bf721d75baa8d534 on tablet 2ed04eb8c46f4cf4bf721d75baa8d534 with peer ec7796d40c8a463caa442aa67ba3d367
I20240502 07:58:36.875159 22856 raft_consensus.cc:1050] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2: attempting to promote NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 to VOTER
I20240502 07:58:36.872651 23761 ts_tablet_manager.cc:1875] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:36.875579 22856 consensus_queue.cc:237] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } }
I20240502 07:58:36.879565 21668 raft_consensus.cc:1261] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240502 07:58:36.880064 23459 consensus_queue.cc:1035] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20240502 07:58:36.880152 23788 ts_tablet_manager.cc:1875] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:36.882196 23209 raft_consensus.cc:1261] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240502 07:58:36.883005 23459 consensus_queue.cc:1035] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20240502 07:58:36.888523 23747 tablet_bootstrap.cc:492] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6: Bootstrap replayed 1/1 log segments. Stats: ops{read=61 overwritten=0 applied=61 ignored=0} inserts{seen=746 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:36.889173 23747 tablet_bootstrap.cc:492] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6: Bootstrap complete.
I20240502 07:58:36.889438 23747 ts_tablet_manager.cc:1367] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6: Time spent bootstrapping tablet: real 0.256s user 0.073s sys 0.000s
I20240502 07:58:36.889927 23747 raft_consensus.cc:348] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } }
I20240502 07:58:36.890064 23747 raft_consensus.cc:729] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c0ee0b72cadc4ebaa2e32a046a107ce6, State: Initialized, Role: LEARNER
I20240502 07:58:36.890182 23747 consensus_queue.cc:260] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 61, Last appended: 1.61, Last appended by leader: 61, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } }
I20240502 07:58:36.890552 23747 ts_tablet_manager.cc:1398] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:36.891005 21692 tablet_copy_service.cc:342] P e218451f770e4ab0a5266e60bdb30a6a: Request end of tablet copy session c0ee0b72cadc4ebaa2e32a046a107ce6-e46a300254bb45d1970dfb5db271ad14 received from {username='slave'} at 127.18.13.134:58139
I20240502 07:58:36.891089 21692 tablet_copy_service.cc:433] P e218451f770e4ab0a5266e60bdb30a6a: ending tablet copy session c0ee0b72cadc4ebaa2e32a046a107ce6-e46a300254bb45d1970dfb5db271ad14 on tablet e46a300254bb45d1970dfb5db271ad14 with peer c0ee0b72cadc4ebaa2e32a046a107ce6
I20240502 07:58:36.890945 23459 raft_consensus.cc:2932] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index 44 to 72, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:36.896768 23788 ts_tablet_manager.cc:1888] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.84
I20240502 07:58:36.896891 23788 log.cc:1196] T 829291de88d84d6b8ead3a450430acba P e218451f770e4ab0a5266e60bdb30a6a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/829291de88d84d6b8ead3a450430acba
I20240502 07:58:36.898007 21668 raft_consensus.cc:2932] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 44 to 72, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:36.900552 21523 catalog_manager.cc:5484] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index 44 to 72, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:36.916254 23209 raft_consensus.cc:2932] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 44 to 72, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:36.916800 23761 ts_tablet_manager.cc:1888] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.71
I20240502 07:58:36.916891 23761 log.cc:1196] T e0c289e06aba4976ab7249d72f26c0e0 P 21b4d9bd384846b2ad74c199585018ba: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/e0c289e06aba4976ab7249d72f26c0e0
I20240502 07:58:36.917505 21517 catalog_manager.cc:4830] TS 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811): tablet e0c289e06aba4976ab7249d72f26c0e0 (table rebalance_test_table_3 [id=aba27c7bd3a84e94b7e627dc4e8d5c6a]) successfully deleted
I20240502 07:58:36.922062 21515 catalog_manager.cc:4830] TS e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667): tablet 829291de88d84d6b8ead3a450430acba (table rebalance_test_table_3 [id=aba27c7bd3a84e94b7e627dc4e8d5c6a]) successfully deleted
I20240502 07:58:36.929630 23672 raft_consensus.cc:1050] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2: attempting to promote NON_VOTER ec7796d40c8a463caa442aa67ba3d367 to VOTER
I20240502 07:58:36.930058 23672 consensus_queue.cc:237] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 1.59, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } }
I20240502 07:58:36.931699 22947 raft_consensus.cc:1261] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch)
I20240502 07:58:36.932210 21933 raft_consensus.cc:1261] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch)
I20240502 07:58:36.932714 23413 consensus_queue.cc:1035] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 58, Time since last communication: 0.000s
I20240502 07:58:36.933007 23413 consensus_queue.cc:1035] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s
I20240502 07:58:36.935181 23413 raft_consensus.cc:2932] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.60: config changed from index 41 to 60, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:36.937706 21934 raft_consensus.cc:2932] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 41 to 60, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:36.938174 22946 raft_consensus.cc:2932] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 41 to 60, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:36.940564 21526 catalog_manager.cc:5484] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba reported cstate change: config changed from index 41 to 60, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:36.950462 21803 consensus_queue.cc:237] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 1.72, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } }
I20240502 07:58:36.952759 23209 raft_consensus.cc:1261] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 73. (index mismatch)
I20240502 07:58:36.953552 23436 consensus_queue.cc:1035] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20240502 07:58:36.954573 21803 consensus_queue.cc:237] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 1.62, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } }
I20240502 07:58:36.956312 22947 raft_consensus.cc:1261] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 63. (index mismatch)
I20240502 07:58:36.963196 23413 consensus_queue.cc:1035] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 61, Time since last communication: 0.000s
I20240502 07:58:36.965445 23413 raft_consensus.cc:2932] T c725ad05375344c3b52682bc91090a43 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.63: config changed from index 60 to 63, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:36.966686 23715 tablet_bootstrap.cc:492] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367: Bootstrap replayed 1/1 log segments. Stats: ops{read=53 overwritten=0 applied=53 ignored=0} inserts{seen=684 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:36.967005 23715 tablet_bootstrap.cc:492] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367: Bootstrap complete.
I20240502 07:58:36.967197 23715 ts_tablet_manager.cc:1367] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367: Time spent bootstrapping tablet: real 0.135s user 0.060s sys 0.000s
I20240502 07:58:36.967309 22946 raft_consensus.cc:2932] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Committing config change with OpId 1.63: config changed from index 60 to 63, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:36.967620 23715 raft_consensus.cc:348] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } }
I20240502 07:58:36.967730 23715 raft_consensus.cc:729] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ec7796d40c8a463caa442aa67ba3d367, State: Initialized, Role: LEARNER
I20240502 07:58:36.967852 23715 consensus_queue.cc:260] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 53, Last appended: 1.53, Last appended by leader: 53, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 52 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: true } }
I20240502 07:58:36.970808 21529 catalog_manager.cc:5484] T c725ad05375344c3b52682bc91090a43 P ec7796d40c8a463caa442aa67ba3d367 reported cstate change: config changed from index 60 to 63, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:36.973413 21913 tablet_service.cc:1514] Processing DeleteTablet for tablet c725ad05375344c3b52682bc91090a43 with delete_type TABLET_DATA_TOMBSTONED (TS 21b4d9bd384846b2ad74c199585018ba not found in new config with opid_index 63) from {username='slave'} at 127.0.0.1:40950
I20240502 07:58:36.973626 23715 ts_tablet_manager.cc:1398] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 07:58:36.977497 23761 tablet_replica.cc:330] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba: stopping tablet replica
I20240502 07:58:36.977699 23761 raft_consensus.cc:2229] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:36.977828 23761 raft_consensus.cc:2258] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:36.978926 21515 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c725ad05375344c3b52682bc91090a43 with cas_config_opid_index 60: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:36.980722 21823 tablet_copy_service.cc:342] P e8406d5a62054258be7fff420a4a4fe2: Request end of tablet copy session ec7796d40c8a463caa442aa67ba3d367-d59002c98d3f4ab8bbea19caaf0c8419 received from {username='slave'} at 127.18.13.132:37655
I20240502 07:58:36.980823 21823 tablet_copy_service.cc:433] P e8406d5a62054258be7fff420a4a4fe2: ending tablet copy session ec7796d40c8a463caa442aa67ba3d367-d59002c98d3f4ab8bbea19caaf0c8419 on tablet d59002c98d3f4ab8bbea19caaf0c8419 with peer ec7796d40c8a463caa442aa67ba3d367
I20240502 07:58:36.981330 23672 raft_consensus.cc:2932] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.73: config changed from index 72 to 73, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:36.989212 22946 raft_consensus.cc:1203] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 07:58:36.992905 23209 raft_consensus.cc:2932] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 72 to 73, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:36.994333 21515 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet efe71ca9e2604316b7b93385d4a9eace with cas_config_opid_index 72: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:36.997037 21529 catalog_manager.cc:5484] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 reported cstate change: config changed from index 72 to 73, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:37.001494 21651 tablet_service.cc:1514] Processing DeleteTablet for tablet efe71ca9e2604316b7b93385d4a9eace with delete_type TABLET_DATA_TOMBSTONED (TS e218451f770e4ab0a5266e60bdb30a6a not found in new config with opid_index 73) from {username='slave'} at 127.0.0.1:46142
I20240502 07:58:37.003559 23788 tablet_replica.cc:330] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a: stopping tablet replica
I20240502 07:58:37.005182 23617 raft_consensus.cc:1050] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba: attempting to promote NON_VOTER f6079903ad93465c8741bf16583fe8c8 to VOTER
I20240502 07:58:37.005544 23617 consensus_queue.cc:237] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 1.64, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } }
I20240502 07:58:37.007005 23761 ts_tablet_manager.cc:1875] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:37.007757 21671 raft_consensus.cc:1261] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240502 07:58:37.007908 23077 raft_consensus.cc:1261] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240502 07:58:37.009140 22752 consensus_queue.cc:1035] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20240502 07:58:37.009505 22752 consensus_queue.cc:1035] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20240502 07:58:37.012859 22752 raft_consensus.cc:2932] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.65: config changed from index 35 to 65, f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:37.013463 23761 ts_tablet_manager.cc:1888] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.62
I20240502 07:58:37.013545 23761 log.cc:1196] T c725ad05375344c3b52682bc91090a43 P 21b4d9bd384846b2ad74c199585018ba: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/c725ad05375344c3b52682bc91090a43
I20240502 07:58:37.015256 21671 raft_consensus.cc:2932] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 35 to 65, f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:37.015992 21517 catalog_manager.cc:4830] TS 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811): tablet c725ad05375344c3b52682bc91090a43 (table rebalance_test_table_1 [id=5d61f2326983407fa0ce10f3bd2dcf48]) successfully deleted
I20240502 07:58:37.016094 21529 catalog_manager.cc:5484] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba reported cstate change: config changed from index 35 to 65, f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:37.024498 23078 raft_consensus.cc:2932] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 35 to 65, f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:37.027494 21934 consensus_queue.cc:237] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } }
I20240502 07:58:37.056644 23076 raft_consensus.cc:1261] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8 [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240502 07:58:37.057407 23449 consensus_queue.cc:1035] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240502 07:58:37.059906 23449 raft_consensus.cc:2932] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index 65 to 66, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:37.061616 21517 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f3f58339fd2c4528a2decb910ea73e21 with cas_config_opid_index 65: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:37.062289 23077 raft_consensus.cc:2932] T f3f58339fd2c4528a2decb910ea73e21 P f6079903ad93465c8741bf16583fe8c8 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 65 to 66, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:37.062592 21526 catalog_manager.cc:5484] T f3f58339fd2c4528a2decb910ea73e21 P 21b4d9bd384846b2ad74c199585018ba reported cstate change: config changed from index 65 to 66, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:37.067983 21651 tablet_service.cc:1514] Processing DeleteTablet for tablet f3f58339fd2c4528a2decb910ea73e21 with delete_type TABLET_DATA_TOMBSTONED (TS e218451f770e4ab0a5266e60bdb30a6a not found in new config with opid_index 66) from {username='slave'} at 127.0.0.1:46142
I20240502 07:58:37.079530 23788 raft_consensus.cc:2229] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:37.083465 23788 raft_consensus.cc:2258] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:37.089058 23788 ts_tablet_manager.cc:1875] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:37.093540 23209 raft_consensus.cc:1203] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.38->[1.39-1.64] Dedup: 1.52->[1.53-1.64]
I20240502 07:58:37.106840 23788 ts_tablet_manager.cc:1888] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.72
I20240502 07:58:37.114005 23788 log.cc:1196] T efe71ca9e2604316b7b93385d4a9eace P e218451f770e4ab0a5266e60bdb30a6a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/efe71ca9e2604316b7b93385d4a9eace
I20240502 07:58:37.114665 21515 catalog_manager.cc:4830] TS e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667): tablet efe71ca9e2604316b7b93385d4a9eace (table rebalance_test_table_4 [id=83a6a35d6fa24f97bab7a6a64be5fe51]) successfully deleted
I20240502 07:58:37.119005 23788 tablet_replica.cc:330] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a: stopping tablet replica
I20240502 07:58:37.119278 23788 raft_consensus.cc:2229] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:37.119508 23788 raft_consensus.cc:2258] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:37.120806 23788 ts_tablet_manager.cc:1875] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:37.141147 23209 raft_consensus.cc:1203] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.43->[1.44-1.80] Dedup: 1.61->[1.62-1.80]
I20240502 07:58:37.160748 23788 ts_tablet_manager.cc:1888] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.65
I20240502 07:58:37.161417 23788 log.cc:1196] T f3f58339fd2c4528a2decb910ea73e21 P e218451f770e4ab0a5266e60bdb30a6a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/f3f58339fd2c4528a2decb910ea73e21
I20240502 07:58:37.164245 21515 catalog_manager.cc:4830] TS e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667): tablet f3f58339fd2c4528a2decb910ea73e21 (table rebalance_test_table_0 [id=b1571debd96c4d31a1be20579ff6bb36]) successfully deleted
I20240502 07:58:37.172008 23436 raft_consensus.cc:1050] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2: attempting to promote NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 to VOTER
I20240502 07:58:37.172459 23436 consensus_queue.cc:237] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } }
I20240502 07:58:37.174830 21668 raft_consensus.cc:1261] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240502 07:58:37.175227 23208 raft_consensus.cc:1261] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240502 07:58:37.178197 23413 consensus_queue.cc:1035] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 64, Time since last communication: 0.000s
I20240502 07:58:37.178547 23413 consensus_queue.cc:1035] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240502 07:58:37.181152 23413 raft_consensus.cc:2932] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index 39 to 66, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:37.181593 21668 raft_consensus.cc:2932] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 39 to 66, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:37.185300 21526 catalog_manager.cc:5484] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: config changed from index 39 to 66, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:37.188993 23208 raft_consensus.cc:2932] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 39 to 66, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:37.194438 21803 consensus_queue.cc:237] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 1.66, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } }
I20240502 07:58:37.199296 22946 raft_consensus.cc:1203] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.51->[1.52-1.62] Dedup: 1.53->[1.54-1.62]
I20240502 07:58:37.201045 23208 raft_consensus.cc:1261] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240502 07:58:37.203903 23436 consensus_queue.cc:1035] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20240502 07:58:37.216982 23413 raft_consensus.cc:1050] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2: attempting to promote NON_VOTER ec7796d40c8a463caa442aa67ba3d367 to VOTER
I20240502 07:58:37.217404 23413 consensus_queue.cc:237] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } }
I20240502 07:58:37.219503 23436 raft_consensus.cc:2932] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.67: config changed from index 66 to 67, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:37.221024 21515 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0029538492ea4f25a822f18de9c83c54 with cas_config_opid_index 66: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:37.221240 21934 raft_consensus.cc:1261] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240502 07:58:37.221926 23208 raft_consensus.cc:2932] T 0029538492ea4f25a822f18de9c83c54 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index 66 to 67, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:37.222719 21526 catalog_manager.cc:5484] T 0029538492ea4f25a822f18de9c83c54 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index 66 to 67, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:37.225346 21651 tablet_service.cc:1514] Processing DeleteTablet for tablet 0029538492ea4f25a822f18de9c83c54 with delete_type TABLET_DATA_TOMBSTONED (TS e218451f770e4ab0a5266e60bdb30a6a not found in new config with opid_index 67) from {username='slave'} at 127.0.0.1:46142
I20240502 07:58:37.225726 22946 raft_consensus.cc:1261] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240502 07:58:37.225939 23788 tablet_replica.cc:330] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a: stopping tablet replica
I20240502 07:58:37.226078 23788 raft_consensus.cc:2229] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:37.226204 23788 raft_consensus.cc:2258] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:37.226372 23842 consensus_queue.cc:1035] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20240502 07:58:37.227671 23788 ts_tablet_manager.cc:1875] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:37.230818 23436 consensus_queue.cc:1035] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20240502 07:58:37.232937 23673 raft_consensus.cc:2932] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index 52 to 64, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:37.233373 22947 raft_consensus.cc:2932] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 52 to 64, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:37.237861 21934 raft_consensus.cc:2932] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 52 to 64, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:37.238457 21529 catalog_manager.cc:5484] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index 52 to 64, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:37.242936 21803 consensus_queue.cc:237] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } }
I20240502 07:58:37.244565 22947 raft_consensus.cc:1261] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240502 07:58:37.231772 23788 ts_tablet_manager.cc:1888] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.66
I20240502 07:58:37.245388 23788 log.cc:1196] T 0029538492ea4f25a822f18de9c83c54 P e218451f770e4ab0a5266e60bdb30a6a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/0029538492ea4f25a822f18de9c83c54
I20240502 07:58:37.246086 21515 catalog_manager.cc:4830] TS e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667): tablet 0029538492ea4f25a822f18de9c83c54 (table rebalance_test_table_0 [id=b1571debd96c4d31a1be20579ff6bb36]) successfully deleted
I20240502 07:58:37.246620 23413 consensus_queue.cc:1035] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240502 07:58:37.253154 22856 raft_consensus.cc:2932] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index 64 to 66, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:37.253947 23769 raft_consensus.cc:1050] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a: attempting to promote NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 to VOTER
I20240502 07:58:37.254307 23769 consensus_queue.cc:237] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } }
I20240502 07:58:37.255657 21515 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d59002c98d3f4ab8bbea19caaf0c8419 with cas_config_opid_index 64: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:37.256148 23207 raft_consensus.cc:1261] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240502 07:58:37.256526 23468 consensus_queue.cc:1035] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 82, Time since last communication: 0.000s
I20240502 07:58:37.258028 21934 raft_consensus.cc:1261] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240502 07:58:37.258759 21526 catalog_manager.cc:5484] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index 64 to 66, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:37.260241 23468 consensus_queue.cc:1035] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240502 07:58:37.262735 23427 raft_consensus.cc:2932] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index 44 to 84, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:37.263887 21913 tablet_service.cc:1514] Processing DeleteTablet for tablet d59002c98d3f4ab8bbea19caaf0c8419 with delete_type TABLET_DATA_TOMBSTONED (TS 21b4d9bd384846b2ad74c199585018ba not found in new config with opid_index 66) from {username='slave'} at 127.0.0.1:40950
I20240502 07:58:37.264115 23761 tablet_replica.cc:330] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba: stopping tablet replica
I20240502 07:58:37.264271 23761 raft_consensus.cc:2229] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:37.264515 23761 raft_consensus.cc:2258] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:37.265409 21931 raft_consensus.cc:2932] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 44 to 84, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:37.265430 23761 ts_tablet_manager.cc:1875] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:37.267246 21529 catalog_manager.cc:5484] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: config changed from index 44 to 84, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:37.270011 23761 ts_tablet_manager.cc:1888] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.65
I20240502 07:58:37.270396 23761 log.cc:1196] T d59002c98d3f4ab8bbea19caaf0c8419 P 21b4d9bd384846b2ad74c199585018ba: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/d59002c98d3f4ab8bbea19caaf0c8419
I20240502 07:58:37.270205 22946 raft_consensus.cc:2932] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 64 to 66, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:37.271302 21517 catalog_manager.cc:4830] TS 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811): tablet d59002c98d3f4ab8bbea19caaf0c8419 (table rebalance_test_table_4 [id=83a6a35d6fa24f97bab7a6a64be5fe51]) successfully deleted
I20240502 07:58:37.279204 23208 raft_consensus.cc:2932] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 44 to 84, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:37.448194 22947 raft_consensus.cc:1203] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.40->[1.41-1.80] Dedup: 1.64->[1.65-1.80]
I20240502 07:58:37.481851 22856 raft_consensus.cc:1050] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2: attempting to promote NON_VOTER ec7796d40c8a463caa442aa67ba3d367 to VOTER
I20240502 07:58:37.482175 22856 consensus_queue.cc:237] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } }
I20240502 07:58:37.484172 22945 raft_consensus.cc:1261] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240502 07:58:37.485424 23673 consensus_queue.cc:1035] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 81, Time since last communication: 0.000s
I20240502 07:58:37.487998 23459 raft_consensus.cc:2932] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index 41 to 83, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:37.489996 22947 raft_consensus.cc:2932] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index 41 to 83, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:37.492396 21526 catalog_manager.cc:5484] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index 41 to 83, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:37.498562 21669 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e46a300254bb45d1970dfb5db271ad14"
dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36966
I20240502 07:58:37.498754 21669 raft_consensus.cc:595] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Received request to transfer leadership
I20240502 07:58:37.499428 23475 raft_consensus.cc:979] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a: : Instructing follower c0ee0b72cadc4ebaa2e32a046a107ce6 to start an election
I20240502 07:58:37.499517 23475 raft_consensus.cc:1067] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Signalling peer c0ee0b72cadc4ebaa2e32a046a107ce6 to start an election
I20240502 07:58:37.500871 23209 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e46a300254bb45d1970dfb5db271ad14"
dest_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6"
from {username='slave'} at 127.18.13.129:55041
I20240502 07:58:37.501036 23209 raft_consensus.cc:482] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:58:37.501122 23209 raft_consensus.cc:3037] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:37.502199 21671 raft_consensus.cc:1261] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240502 07:58:37.502200 23209 raft_consensus.cc:504] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } }
I20240502 07:58:37.502821 23672 consensus_queue.cc:1035] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 81, Time since last communication: 0.000s
I20240502 07:58:37.503592 21668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46a300254bb45d1970dfb5db271ad14" candidate_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" candidate_term: 2 candidate_status { last_received { term: 1 index: 95 } } ignore_live_leader: true dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a"
I20240502 07:58:37.503698 21668 raft_consensus.cc:3032] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:58:37.503736 21668 raft_consensus.cc:729] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Running, Role: LEADER
I20240502 07:58:37.503896 21668 consensus_queue.cc:260] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 95, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } }
I20240502 07:58:37.504069 21668 raft_consensus.cc:3037] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:37.504788 23209 leader_election.cc:288] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [CANDIDATE]: Term 2 election: Requested vote from peers e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667), 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:37.506145 22752 raft_consensus.cc:1050] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba: attempting to promote NON_VOTER ec7796d40c8a463caa442aa67ba3d367 to VOTER
I20240502 07:58:37.506487 22752 consensus_queue.cc:237] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } }
I20240502 07:58:37.509142 21669 raft_consensus.cc:2932] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index 41 to 83, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:37.511078 22946 raft_consensus.cc:1261] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367 [term 1 LEARNER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 07:58:37.511991 23484 consensus_queue.cc:1035] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 07:58:37.516116 21668 raft_consensus.cc:2454] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ee0b72cadc4ebaa2e32a046a107ce6 in term 2.
I20240502 07:58:37.518536 23143 leader_election.cc:302] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ee0b72cadc4ebaa2e32a046a107ce6, e218451f770e4ab0a5266e60bdb30a6a; no voters:
I20240502 07:58:37.518783 23770 raft_consensus.cc:2781] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:37.518982 23770 raft_consensus.cc:686] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 2 LEADER]: Becoming Leader. State: Replica: c0ee0b72cadc4ebaa2e32a046a107ce6, State: Running, Role: LEADER
I20240502 07:58:37.519313 23770 consensus_queue.cc:237] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 95, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } }
I20240502 07:58:37.522089 21803 raft_consensus.cc:1261] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 07:58:37.523077 23484 consensus_queue.cc:1035] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 07:58:37.526592 21526 catalog_manager.cc:5484] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 reported cstate change: term changed from 1 to 2, leader changed from e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) to c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134). New cstate: current_term: 2 leader_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:37.528817 23617 raft_consensus.cc:2932] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:37.536118 21803 raft_consensus.cc:2932] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:37.536204 21526 catalog_manager.cc:5484] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba reported cstate change: config changed from index 2 to 3, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:37.566573 22946 raft_consensus.cc:2932] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:37.569294 21931 consensus_queue.cc:237] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } }
I20240502 07:58:37.570924 22946 raft_consensus.cc:1261] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:58:37.571429 23449 consensus_queue.cc:1035] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:58:37.579039 21630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36818: Illegal state: replica e218451f770e4ab0a5266e60bdb30a6a is not leader of this config: current role FOLLOWER
I20240502 07:58:37.580641 21934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46a300254bb45d1970dfb5db271ad14" candidate_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" candidate_term: 2 candidate_status { last_received { term: 1 index: 95 } } ignore_live_leader: true dest_uuid: "21b4d9bd384846b2ad74c199585018ba"
I20240502 07:58:37.580772 21934 raft_consensus.cc:3037] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:37.581779 21934 raft_consensus.cc:2454] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ee0b72cadc4ebaa2e32a046a107ce6 in term 2.
I20240502 07:58:37.591610 23449 raft_consensus.cc:2932] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:37.593327 21517 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 01e44d65c0314f0a9d30a2587723c290 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:37.594115 22946 raft_consensus.cc:2932] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:37.594482 21529 catalog_manager.cc:5484] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba reported cstate change: config changed from index 3 to 4, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:37.598953 21782 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e44d65c0314f0a9d30a2587723c290 with delete_type TABLET_DATA_TOMBSTONED (TS e8406d5a62054258be7fff420a4a4fe2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51132
I20240502 07:58:37.610700 23900 tablet_replica.cc:330] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2: stopping tablet replica
W20240502 07:58:37.613834 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51072: Illegal state: replica 21b4d9bd384846b2ad74c199585018ba is not leader of this config: current role FOLLOWER
I20240502 07:58:37.614364 23900 raft_consensus.cc:2229] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:37.614567 23900 raft_consensus.cc:2258] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:37.626358 23900 ts_tablet_manager.cc:1875] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:37.628931 23900 ts_tablet_manager.cc:1888] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 07:58:37.629029 23900 log.cc:1196] T 01e44d65c0314f0a9d30a2587723c290 P e8406d5a62054258be7fff420a4a4fe2: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/01e44d65c0314f0a9d30a2587723c290
I20240502 07:58:37.631682 21515 catalog_manager.cc:4830] TS e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891): tablet 01e44d65c0314f0a9d30a2587723c290 (table TestTable [id=828c706eb9dd489bb081a01b553ce7ff]) successfully deleted
I20240502 07:58:37.632628 21803 consensus_queue.cc:237] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } }
I20240502 07:58:37.634018 21671 raft_consensus.cc:1261] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240502 07:58:37.634377 21931 raft_consensus.cc:1261] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [term 2 FOLLOWER]: Refusing update from remote peer c0ee0b72cadc4ebaa2e32a046a107ce6: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 2 index: 97. (index mismatch)
I20240502 07:58:37.635082 23673 consensus_queue.cc:1035] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.000s
I20240502 07:58:37.635550 23770 consensus_queue.cc:1035] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240502 07:58:37.637667 23673 raft_consensus.cc:2932] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index -1 to 94, NON_VOTER f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } } }
I20240502 07:58:37.638497 21668 raft_consensus.cc:1261] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 2 FOLLOWER]: Refusing update from remote peer c0ee0b72cadc4ebaa2e32a046a107ce6: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 2 index: 97. (index mismatch)
I20240502 07:58:37.639508 23770 consensus_queue.cc:1035] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240502 07:58:37.641979 21671 raft_consensus.cc:2932] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index -1 to 94, NON_VOTER f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } } }
I20240502 07:58:37.642724 21526 catalog_manager.cc:5484] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index -1 to 94, NON_VOTER f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:37.642894 23839 mvcc.cc:205] Tried to move back new op lower bound from 7023151995421011968 to 7023151994962255872. Current Snapshot: MvccSnapshot[applied={T|T < 7023151995421011968}]
W20240502 07:58:37.650707 21736 consensus_peers.cc:485] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 -> Peer f97f8632bbe64638bb1ad2679849f434 (127.18.13.135:43407): Couldn't send request to peer f97f8632bbe64638bb1ad2679849f434. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e956783425bb49d2ab78be427e61781f. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:37.686234 23206 consensus_queue.cc:237] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 2.100, Last appended by leader: 95, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } }
I20240502 07:58:37.688699 21931 raft_consensus.cc:1261] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [term 2 FOLLOWER]: Refusing update from remote peer c0ee0b72cadc4ebaa2e32a046a107ce6: Log matching property violated. Preceding OpId in replica: term: 2 index: 99. Preceding OpId from leader: term: 2 index: 101. (index mismatch)
I20240502 07:58:37.689221 23770 consensus_queue.cc:1035] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 99, Time since last communication: 0.000s
I20240502 07:58:37.695479 23770 raft_consensus.cc:2932] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 2 LEADER]: Committing config change with OpId 2.101: config changed from index 84 to 101, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New config: { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:37.696475 21931 raft_consensus.cc:2932] T e46a300254bb45d1970dfb5db271ad14 P 21b4d9bd384846b2ad74c199585018ba [term 2 FOLLOWER]: Committing config change with OpId 2.101: config changed from index 84 to 101, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New config: { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:37.698035 21515 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e46a300254bb45d1970dfb5db271ad14 with cas_config_opid_index 84: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:37.698078 21529 catalog_manager.cc:5484] T e46a300254bb45d1970dfb5db271ad14 P c0ee0b72cadc4ebaa2e32a046a107ce6 reported cstate change: config changed from index 84 to 101, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New cstate: current_term: 2 leader_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" committed_config { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:37.703816 21651 tablet_service.cc:1514] Processing DeleteTablet for tablet e46a300254bb45d1970dfb5db271ad14 with delete_type TABLET_DATA_TOMBSTONED (TS e218451f770e4ab0a5266e60bdb30a6a not found in new config with opid_index 101) from {username='slave'} at 127.0.0.1:46142
I20240502 07:58:37.704061 23788 tablet_replica.cc:330] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a: stopping tablet replica
I20240502 07:58:37.704206 23788 raft_consensus.cc:2229] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:37.704300 23788 pending_rounds.cc:62] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a: Trying to abort 1 pending ops.
I20240502 07:58:37.704334 23788 pending_rounds.cc:68] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a: Aborting op as it isn't in flight: id { term: 2 index: 100 } timestamp: 7023151995589738496 op_type: WRITE_OP write_request { tablet_id: "e46a300254bb45d1970dfb5db271ad14" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\334\232\254\007\223\014~\211\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""59526488e08c9510" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023151995547316224 authz_token { token_data: "\010\326\221\315\261\006\"3\n\005slave\022*\n 5d61f2326983407fa0ce10f3bd2dcf48\020\001\030\001 \001(\001" signature: "<redacted>""\032\256F\0360\032\t\033:\337^K\242\240\305\364\006\317K`\331>\344\205\342\260/\270`>\3558`\2229G;\371\'[\267\343M:\033\'\330\267O\nE\2338\257\010\007wi\267\263\326\316%\036" signing_key_seq_num: 0 } } request_id { client_id: "8b70c325b8134002a9da01ca7d556abf" seq_no: 243 first_incomplete_seq_no: 243 attempt_no: 0 }
I20240502 07:58:37.704629 23788 raft_consensus.cc:2258] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:37.705556 23788 ts_tablet_manager.cc:1875] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:37.715211 23788 ts_tablet_manager.cc:1888] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.100
I20240502 07:58:37.715306 23788 log.cc:1196] T e46a300254bb45d1970dfb5db271ad14 P e218451f770e4ab0a5266e60bdb30a6a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/e46a300254bb45d1970dfb5db271ad14
I20240502 07:58:37.715864 21515 catalog_manager.cc:4830] TS e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667): tablet e46a300254bb45d1970dfb5db271ad14 (table rebalance_test_table_1 [id=5d61f2326983407fa0ce10f3bd2dcf48]) successfully deleted
I20240502 07:58:37.739698 21803 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ed04eb8c46f4cf4bf721d75baa8d534"
dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55680
I20240502 07:58:37.739939 21803 raft_consensus.cc:595] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Received request to transfer leadership
W20240502 07:58:37.779600 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:37.784879 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:37.791394 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:37.800055 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:37.807375 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
I20240502 07:58:37.814396 21803 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ed04eb8c46f4cf4bf721d75baa8d534"
dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55714
I20240502 07:58:37.814549 21803 raft_consensus.cc:595] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Received request to transfer leadership
W20240502 07:58:37.818181 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:37.828792 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:37.841521 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:37.854028 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:37.870157 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:37.887009 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:37.900463 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:37.918007 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:37.932219 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:37.949573 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:37.967717 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:37.986318 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:38.007563 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:38.028077 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:38.050702 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
I20240502 07:58:38.069803 21803 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ed04eb8c46f4cf4bf721d75baa8d534"
dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55746
I20240502 07:58:38.069965 21803 raft_consensus.cc:595] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Received request to transfer leadership
W20240502 07:58:38.076570 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:38.102819 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:38.129585 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:38.154397 21762 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
W20240502 07:58:38.183833 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Service unavailable: leader transfer in progress
I20240502 07:58:38.189345 23436 raft_consensus.cc:979] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2: : Instructing follower ec7796d40c8a463caa442aa67ba3d367 to start an election
I20240502 07:58:38.189481 23673 raft_consensus.cc:1067] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Signalling peer ec7796d40c8a463caa442aa67ba3d367 to start an election
I20240502 07:58:38.190172 22946 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ed04eb8c46f4cf4bf721d75baa8d534"
dest_uuid: "ec7796d40c8a463caa442aa67ba3d367"
from {username='slave'} at 127.18.13.130:33903
I20240502 07:58:38.190351 22946 raft_consensus.cc:482] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:58:38.190465 22946 raft_consensus.cc:3037] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:38.191498 22946 raft_consensus.cc:504] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } }
I20240502 07:58:38.192803 22946 leader_election.cc:288] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [CANDIDATE]: Term 2 election: Requested vote from peers e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891), e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:38.194443 21671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed04eb8c46f4cf4bf721d75baa8d534" candidate_uuid: "ec7796d40c8a463caa442aa67ba3d367" candidate_term: 2 candidate_status { last_received { term: 1 index: 89 } } ignore_live_leader: true dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a"
I20240502 07:58:38.194693 21671 raft_consensus.cc:3037] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:38.195680 21671 raft_consensus.cc:2454] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec7796d40c8a463caa442aa67ba3d367 in term 2.
I20240502 07:58:38.196453 22881 leader_election.cc:302] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e218451f770e4ab0a5266e60bdb30a6a, ec7796d40c8a463caa442aa67ba3d367; no voters:
I20240502 07:58:38.197453 21803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed04eb8c46f4cf4bf721d75baa8d534" candidate_uuid: "ec7796d40c8a463caa442aa67ba3d367" candidate_term: 2 candidate_status { last_received { term: 1 index: 89 } } ignore_live_leader: true dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
I20240502 07:58:38.197583 21803 raft_consensus.cc:3032] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:58:38.197643 21803 raft_consensus.cc:729] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Running, Role: LEADER
I20240502 07:58:38.197815 21803 consensus_queue.cc:260] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 1.89, Last appended by leader: 89, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } }
I20240502 07:58:38.198000 21803 raft_consensus.cc:3037] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:38.199450 21803 raft_consensus.cc:2454] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec7796d40c8a463caa442aa67ba3d367 in term 2.
I20240502 07:58:38.201519 23975 raft_consensus.cc:2781] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:38.201771 23975 raft_consensus.cc:686] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [term 2 LEADER]: Becoming Leader. State: Replica: ec7796d40c8a463caa442aa67ba3d367, State: Running, Role: LEADER
I20240502 07:58:38.202173 23975 consensus_queue.cc:237] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 1.89, Last appended by leader: 89, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } }
I20240502 07:58:38.206034 21526 catalog_manager.cc:5484] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 reported cstate change: term changed from 1 to 2, leader changed from e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) to ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132). New cstate: current_term: 2 leader_uuid: "ec7796d40c8a463caa442aa67ba3d367" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:38.207559 23977 ts_tablet_manager.cc:902] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434: Initiating tablet copy from peer e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
W20240502 07:58:38.211292 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51256: Illegal state: replica e8406d5a62054258be7fff420a4a4fe2 is not leader of this config: current role FOLLOWER
I20240502 07:58:38.208714 23977 tablet_copy_client.cc:308] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:40891
I20240502 07:58:38.236492 21823 tablet_copy_service.cc:140] P e8406d5a62054258be7fff420a4a4fe2: Received BeginTabletCopySession request for tablet e956783425bb49d2ab78be427e61781f from peer f97f8632bbe64638bb1ad2679849f434 ({username='slave'} at 127.18.13.135:34699)
I20240502 07:58:38.236639 21823 tablet_copy_service.cc:161] P e8406d5a62054258be7fff420a4a4fe2: Beginning new tablet copy session on tablet e956783425bb49d2ab78be427e61781f from peer f97f8632bbe64638bb1ad2679849f434 at {username='slave'} at 127.18.13.135:34699: session id = f97f8632bbe64638bb1ad2679849f434-e956783425bb49d2ab78be427e61781f
I20240502 07:58:38.237656 21823 tablet_copy_source_session.cc:205] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 07:58:38.246075 21629 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36978: Illegal state: replica e218451f770e4ab0a5266e60bdb30a6a is not leader of this config: current role FOLLOWER
I20240502 07:58:38.250289 23977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e956783425bb49d2ab78be427e61781f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:38.261085 23977 tablet_copy_client.cc:783] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:38.261282 23977 tablet_copy_client.cc:647] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:38.263453 23977 tablet_copy_client.cc:523] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:38.265023 23977 tablet_bootstrap.cc:492] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434: Bootstrap starting.
I20240502 07:58:38.305527 21803 raft_consensus.cc:1261] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Refusing update from remote peer ec7796d40c8a463caa442aa67ba3d367: Log matching property violated. Preceding OpId in replica: term: 1 index: 89. Preceding OpId from leader: term: 2 index: 91. (index mismatch)
I20240502 07:58:38.306457 23975 consensus_queue.cc:1035] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20240502 07:58:38.307516 21668 raft_consensus.cc:1261] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [term 2 FOLLOWER]: Refusing update from remote peer ec7796d40c8a463caa442aa67ba3d367: Log matching property violated. Preceding OpId in replica: term: 1 index: 89. Preceding OpId from leader: term: 2 index: 91. (index mismatch)
I20240502 07:58:38.308949 23975 consensus_queue.cc:1035] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20240502 07:58:38.313275 23840 mvcc.cc:205] Tried to move back new op lower bound from 7023151998172114944 to 7023151997760790528. Current Snapshot: MvccSnapshot[applied={T|T < 7023151998172114944 or (T in {7023151998172114944})}]
I20240502 07:58:38.375159 22947 consensus_queue.cc:237] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 2.94, Last appended by leader: 89, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } }
I20240502 07:58:38.382671 21668 raft_consensus.cc:1261] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [term 2 FOLLOWER]: Refusing update from remote peer ec7796d40c8a463caa442aa67ba3d367: Log matching property violated. Preceding OpId in replica: term: 2 index: 94. Preceding OpId from leader: term: 2 index: 95. (index mismatch)
I20240502 07:58:38.390604 23976 consensus_queue.cc:1035] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 94, Time since last communication: 0.000s
I20240502 07:58:38.399762 23976 raft_consensus.cc:2932] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 [term 2 LEADER]: Committing config change with OpId 2.95: config changed from index 83 to 95, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:38.402318 21514 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ed04eb8c46f4cf4bf721d75baa8d534 with cas_config_opid_index 83: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:38.403096 21668 raft_consensus.cc:2932] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e218451f770e4ab0a5266e60bdb30a6a [term 2 FOLLOWER]: Committing config change with OpId 2.95: config changed from index 83 to 95, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } }
I20240502 07:58:38.403468 21526 catalog_manager.cc:5484] T 2ed04eb8c46f4cf4bf721d75baa8d534 P ec7796d40c8a463caa442aa67ba3d367 reported cstate change: config changed from index 83 to 95, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New cstate: current_term: 2 leader_uuid: "ec7796d40c8a463caa442aa67ba3d367" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:38.406347 21782 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed04eb8c46f4cf4bf721d75baa8d534 with delete_type TABLET_DATA_TOMBSTONED (TS e8406d5a62054258be7fff420a4a4fe2 not found in new config with opid_index 95) from {username='slave'} at 127.0.0.1:51132
I20240502 07:58:38.410198 24005 tablet_replica.cc:330] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2: stopping tablet replica
I20240502 07:58:38.410449 24005 raft_consensus.cc:2229] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:38.410686 24005 raft_consensus.cc:2258] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:38.411535 24005 ts_tablet_manager.cc:1875] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:38.414842 24005 ts_tablet_manager.cc:1888] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.94
I20240502 07:58:38.415021 24005 log.cc:1196] T 2ed04eb8c46f4cf4bf721d75baa8d534 P e8406d5a62054258be7fff420a4a4fe2: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/2ed04eb8c46f4cf4bf721d75baa8d534
I20240502 07:58:38.418133 21515 catalog_manager.cc:4830] TS e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891): tablet 2ed04eb8c46f4cf4bf721d75baa8d534 (table rebalance_test_table_0 [id=b1571debd96c4d31a1be20579ff6bb36]) successfully deleted
I20240502 07:58:38.484913 23977 tablet_bootstrap.cc:492] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434: Bootstrap replayed 1/1 log segments. Stats: ops{read=118 overwritten=0 applied=117 ignored=0} inserts{seen=789 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 07:58:38.485297 23977 tablet_bootstrap.cc:492] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434: Bootstrap complete.
I20240502 07:58:38.485528 23977 ts_tablet_manager.cc:1367] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434: Time spent bootstrapping tablet: real 0.221s user 0.104s sys 0.004s
I20240502 07:58:38.485994 23977 raft_consensus.cc:348] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } }
I20240502 07:58:38.486608 23977 raft_consensus.cc:729] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f97f8632bbe64638bb1ad2679849f434, State: Initialized, Role: LEARNER
I20240502 07:58:38.488022 23977 consensus_queue.cc:260] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 117, Last appended: 1.118, Last appended by leader: 118, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: true } }
I20240502 07:58:38.488458 23977 ts_tablet_manager.cc:1398] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 07:58:38.492888 21823 tablet_copy_service.cc:342] P e8406d5a62054258be7fff420a4a4fe2: Request end of tablet copy session f97f8632bbe64638bb1ad2679849f434-e956783425bb49d2ab78be427e61781f received from {username='slave'} at 127.18.13.135:34699
I20240502 07:58:38.492985 21823 tablet_copy_service.cc:433] P e8406d5a62054258be7fff420a4a4fe2: ending tablet copy session f97f8632bbe64638bb1ad2679849f434-e956783425bb49d2ab78be427e61781f on tablet e956783425bb49d2ab78be427e61781f with peer f97f8632bbe64638bb1ad2679849f434
I20240502 07:58:38.582975 23339 raft_consensus.cc:1203] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.93->[1.94-1.128] Dedup: 1.118->[1.119-1.128]
I20240502 07:58:38.601493 23436 raft_consensus.cc:1050] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2: attempting to promote NON_VOTER f97f8632bbe64638bb1ad2679849f434 to VOTER
I20240502 07:58:38.601873 23436 consensus_queue.cc:237] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 1.129, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } }
I20240502 07:58:38.604736 23339 raft_consensus.cc:1261] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 1 LEARNER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240502 07:58:38.605275 23673 consensus_queue.cc:1035] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 128, Time since last communication: 0.000s
I20240502 07:58:38.605784 21671 raft_consensus.cc:1261] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240502 07:58:38.608810 23413 raft_consensus.cc:2932] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.130: config changed from index 94 to 130, f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:38.609864 23673 consensus_queue.cc:1035] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 128, Time since last communication: 0.000s
I20240502 07:58:38.611454 21671 raft_consensus.cc:2932] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index 94 to 130, f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:38.611860 23339 raft_consensus.cc:2932] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index 94 to 130, f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:38.614146 21526 catalog_manager.cc:5484] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: config changed from index 94 to 130, f97f8632bbe64638bb1ad2679849f434 (127.18.13.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:38.636102 21803 consensus_queue.cc:237] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 1.130, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } }
I20240502 07:58:38.638320 23339 raft_consensus.cc:1261] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 130. Preceding OpId from leader: term: 1 index: 131. (index mismatch)
I20240502 07:58:38.639223 23673 consensus_queue.cc:1035] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20240502 07:58:38.644069 23413 raft_consensus.cc:2932] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.131: config changed from index 130 to 131, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:38.646165 21515 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e956783425bb49d2ab78be427e61781f with cas_config_opid_index 130: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:38.647032 21526 catalog_manager.cc:5484] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index 130 to 131, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:38.648888 23339 raft_consensus.cc:2932] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Committing config change with OpId 1.131: config changed from index 130 to 131, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } }
I20240502 07:58:38.651149 21651 tablet_service.cc:1514] Processing DeleteTablet for tablet e956783425bb49d2ab78be427e61781f with delete_type TABLET_DATA_TOMBSTONED (TS e218451f770e4ab0a5266e60bdb30a6a not found in new config with opid_index 131) from {username='slave'} at 127.0.0.1:46142
I20240502 07:58:38.651525 24026 tablet_replica.cc:330] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a: stopping tablet replica
I20240502 07:58:38.651742 24026 raft_consensus.cc:2229] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:38.652213 24026 raft_consensus.cc:2258] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:38.653439 24026 ts_tablet_manager.cc:1875] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:38.656373 24026 ts_tablet_manager.cc:1888] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.130
I20240502 07:58:38.656519 24026 log.cc:1196] T e956783425bb49d2ab78be427e61781f P e218451f770e4ab0a5266e60bdb30a6a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/e956783425bb49d2ab78be427e61781f
I20240502 07:58:38.657254 21515 catalog_manager.cc:4830] TS e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667): tablet e956783425bb49d2ab78be427e61781f (table rebalance_test_table_2 [id=4f8126ed45de428eaf540e5d65bd2c86]) successfully deleted
I20240502 07:58:39.219283 22927 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:39.224272 23189 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:39.233057 21913 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:39.246520 23319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:39.253095 23058 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:39.291611 21651 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:39.313879 21782 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:58:39.404740 21934 consensus_queue.cc:237] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 168, Committed index: 168, Last appended: 1.168, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } }
I20240502 07:58:39.410845 23340 raft_consensus.cc:1261] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 168. Preceding OpId from leader: term: 1 index: 169. (index mismatch)
I20240502 07:58:39.411823 23484 consensus_queue.cc:1035] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 169, Last known committed idx: 168, Time since last communication: 0.000s
I20240502 07:58:39.415091 22752 raft_consensus.cc:2932] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.169: config changed from index 85 to 169, NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) added. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } } }
I20240502 07:58:39.419589 23340 raft_consensus.cc:2932] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Committing config change with OpId 1.169: config changed from index 85 to 169, NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) added. New config: { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } } }
W20240502 07:58:39.423871 21868 consensus_peers.cc:485] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba -> Peer c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134:42113): Couldn't send request to peer c0ee0b72cadc4ebaa2e32a046a107ce6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 829291de88d84d6b8ead3a450430acba. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:39.433908 21529 catalog_manager.cc:5484] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 reported cstate change: config changed from index 85 to 169, NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } } }
I20240502 07:58:39.479555 21803 consensus_queue.cc:237] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 154, Committed index: 154, Last appended: 1.154, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } }
I20240502 07:58:39.481377 23339 raft_consensus.cc:1261] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 154. Preceding OpId from leader: term: 1 index: 155. (index mismatch)
I20240502 07:58:39.481760 23672 consensus_queue.cc:1035] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 155, Last known committed idx: 154, Time since last communication: 0.000s
I20240502 07:58:39.485852 23672 raft_consensus.cc:2932] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.155: config changed from index 131 to 155, NON_VOTER f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) added. New config: { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } } }
I20240502 07:58:39.489684 23339 raft_consensus.cc:2932] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Committing config change with OpId 1.155: config changed from index 131 to 155, NON_VOTER f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) added. New config: { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } } }
I20240502 07:58:39.493366 21522 catalog_manager.cc:5484] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index 131 to 155, NON_VOTER f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:39.511945 21803 consensus_queue.cc:237] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 152, Committed index: 152, Last appended: 1.152, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 153 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } }
I20240502 07:58:39.513456 23209 raft_consensus.cc:1261] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 152. Preceding OpId from leader: term: 1 index: 153. (index mismatch)
I20240502 07:58:39.515609 24081 consensus_queue.cc:1035] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 153, Last known committed idx: 152, Time since last communication: 0.000s
I20240502 07:58:39.517659 23209 raft_consensus.cc:2932] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Committing config change with OpId 1.153: config changed from index 73 to 153, NON_VOTER f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) added. New config: { opid_index: 153 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } } }
I20240502 07:58:39.517740 23436 raft_consensus.cc:2932] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.153: config changed from index 73 to 153, NON_VOTER f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) added. New config: { opid_index: 153 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } } }
I20240502 07:58:39.525185 21529 catalog_manager.cc:5484] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index 73 to 153, NON_VOTER f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 153 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:39.578619 21738 consensus_peers.cc:485] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 -> Peer f6079903ad93465c8741bf16583fe8c8 (127.18.13.133:38189): Couldn't send request to peer f6079903ad93465c8741bf16583fe8c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efe71ca9e2604316b7b93385d4a9eace. This is attempt 1: this message will repeat every 5th retry.
W20240502 07:58:39.579105 21738 consensus_peers.cc:485] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 -> Peer f6079903ad93465c8741bf16583fe8c8 (127.18.13.133:38189): Couldn't send request to peer f6079903ad93465c8741bf16583fe8c8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e956783425bb49d2ab78be427e61781f. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:40.034930 24139 ts_tablet_manager.cc:902] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6: Initiating tablet copy from peer 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:40.037346 24139 tablet_copy_client.cc:308] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:38811
I20240502 07:58:40.037907 21954 tablet_copy_service.cc:140] P 21b4d9bd384846b2ad74c199585018ba: Received BeginTabletCopySession request for tablet 829291de88d84d6b8ead3a450430acba from peer c0ee0b72cadc4ebaa2e32a046a107ce6 ({username='slave'} at 127.18.13.134:48327)
I20240502 07:58:40.038034 21954 tablet_copy_service.cc:161] P 21b4d9bd384846b2ad74c199585018ba: Beginning new tablet copy session on tablet 829291de88d84d6b8ead3a450430acba from peer c0ee0b72cadc4ebaa2e32a046a107ce6 at {username='slave'} at 127.18.13.134:48327: session id = c0ee0b72cadc4ebaa2e32a046a107ce6-829291de88d84d6b8ead3a450430acba
I20240502 07:58:40.039048 21954 tablet_copy_source_session.cc:205] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:40.040786 24144 ts_tablet_manager.cc:902] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8: Initiating tablet copy from peer e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:40.041370 24144 tablet_copy_client.cc:308] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:40891
I20240502 07:58:40.041666 24139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829291de88d84d6b8ead3a450430acba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:40.043951 24139 tablet_copy_client.cc:783] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:40.044097 24139 tablet_copy_client.cc:647] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:40.050168 24139 tablet_copy_client.cc:523] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:40.052588 24139 tablet_bootstrap.cc:492] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6: Bootstrap starting.
I20240502 07:58:40.053498 21823 tablet_copy_service.cc:140] P e8406d5a62054258be7fff420a4a4fe2: Received BeginTabletCopySession request for tablet e956783425bb49d2ab78be427e61781f from peer f6079903ad93465c8741bf16583fe8c8 ({username='slave'} at 127.18.13.133:57389)
I20240502 07:58:40.053623 21823 tablet_copy_service.cc:161] P e8406d5a62054258be7fff420a4a4fe2: Beginning new tablet copy session on tablet e956783425bb49d2ab78be427e61781f from peer f6079903ad93465c8741bf16583fe8c8 at {username='slave'} at 127.18.13.133:57389: session id = f6079903ad93465c8741bf16583fe8c8-e956783425bb49d2ab78be427e61781f
I20240502 07:58:40.054625 21823 tablet_copy_source_session.cc:205] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:40.056869 24144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e956783425bb49d2ab78be427e61781f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:40.059379 24144 tablet_copy_client.cc:783] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:40.059541 24144 tablet_copy_client.cc:647] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:40.074879 24144 tablet_copy_client.cc:523] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:40.078120 24144 tablet_bootstrap.cc:492] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8: Bootstrap starting.
I20240502 07:58:40.084194 24150 ts_tablet_manager.cc:902] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8: Initiating tablet copy from peer e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:40.085027 24150 tablet_copy_client.cc:308] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:40891
I20240502 07:58:40.085541 21823 tablet_copy_service.cc:140] P e8406d5a62054258be7fff420a4a4fe2: Received BeginTabletCopySession request for tablet efe71ca9e2604316b7b93385d4a9eace from peer f6079903ad93465c8741bf16583fe8c8 ({username='slave'} at 127.18.13.133:57389)
I20240502 07:58:40.085660 21823 tablet_copy_service.cc:161] P e8406d5a62054258be7fff420a4a4fe2: Beginning new tablet copy session on tablet efe71ca9e2604316b7b93385d4a9eace from peer f6079903ad93465c8741bf16583fe8c8 at {username='slave'} at 127.18.13.133:57389: session id = f6079903ad93465c8741bf16583fe8c8-efe71ca9e2604316b7b93385d4a9eace
I20240502 07:58:40.086432 21823 tablet_copy_source_session.cc:205] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:40.087288 24150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe71ca9e2604316b7b93385d4a9eace. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:40.089862 24150 tablet_copy_client.cc:783] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:40.090037 24150 tablet_copy_client.cc:647] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:40.112107 24150 tablet_copy_client.cc:523] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:40.114990 24150 tablet_bootstrap.cc:492] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8: Bootstrap starting.
I20240502 07:58:40.417519 24139 tablet_bootstrap.cc:492] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6: Bootstrap replayed 1/1 log segments. Stats: ops{read=189 overwritten=0 applied=189 ignored=0} inserts{seen=871 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:40.418344 24139 tablet_bootstrap.cc:492] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6: Bootstrap complete.
I20240502 07:58:40.418673 24139 ts_tablet_manager.cc:1367] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6: Time spent bootstrapping tablet: real 0.366s user 0.154s sys 0.008s
I20240502 07:58:40.419675 24139 raft_consensus.cc:348] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } }
I20240502 07:58:40.420090 24139 raft_consensus.cc:729] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c0ee0b72cadc4ebaa2e32a046a107ce6, State: Initialized, Role: LEARNER
I20240502 07:58:40.420264 24139 consensus_queue.cc:260] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 189, Last appended: 1.189, Last appended by leader: 189, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 169 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: true } }
I20240502 07:58:40.420780 24139 ts_tablet_manager.cc:1398] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:58:40.421399 21954 tablet_copy_service.cc:342] P 21b4d9bd384846b2ad74c199585018ba: Request end of tablet copy session c0ee0b72cadc4ebaa2e32a046a107ce6-829291de88d84d6b8ead3a450430acba received from {username='slave'} at 127.18.13.134:48327
I20240502 07:58:40.421486 21954 tablet_copy_service.cc:433] P 21b4d9bd384846b2ad74c199585018ba: ending tablet copy session c0ee0b72cadc4ebaa2e32a046a107ce6-829291de88d84d6b8ead3a450430acba on tablet 829291de88d84d6b8ead3a450430acba with peer c0ee0b72cadc4ebaa2e32a046a107ce6
I20240502 07:58:40.448741 24144 tablet_bootstrap.cc:492] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=163 overwritten=0 applied=163 ignored=0} inserts{seen=832 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:40.449214 24144 tablet_bootstrap.cc:492] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8: Bootstrap complete.
I20240502 07:58:40.449486 24144 ts_tablet_manager.cc:1367] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8: Time spent bootstrapping tablet: real 0.372s user 0.115s sys 0.025s
I20240502 07:58:40.450300 24144 raft_consensus.cc:348] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } }
I20240502 07:58:40.450472 24144 raft_consensus.cc:729] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f6079903ad93465c8741bf16583fe8c8, State: Initialized, Role: LEARNER
I20240502 07:58:40.450620 24144 consensus_queue.cc:260] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 163, Last appended: 1.163, Last appended by leader: 163, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 155 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } }
I20240502 07:58:40.451696 24144 ts_tablet_manager.cc:1398] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:40.453563 21823 tablet_copy_service.cc:342] P e8406d5a62054258be7fff420a4a4fe2: Request end of tablet copy session f6079903ad93465c8741bf16583fe8c8-e956783425bb49d2ab78be427e61781f received from {username='slave'} at 127.18.13.133:57389
I20240502 07:58:40.453670 21823 tablet_copy_service.cc:433] P e8406d5a62054258be7fff420a4a4fe2: ending tablet copy session f6079903ad93465c8741bf16583fe8c8-e956783425bb49d2ab78be427e61781f on tablet e956783425bb49d2ab78be427e61781f with peer f6079903ad93465c8741bf16583fe8c8
I20240502 07:58:40.464164 24150 tablet_bootstrap.cc:492] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=175 overwritten=0 applied=175 ignored=0} inserts{seen=844 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:40.464686 24150 tablet_bootstrap.cc:492] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8: Bootstrap complete.
I20240502 07:58:40.465315 24150 ts_tablet_manager.cc:1367] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8: Time spent bootstrapping tablet: real 0.350s user 0.141s sys 0.008s
I20240502 07:58:40.465857 24150 raft_consensus.cc:348] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 153 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } }
I20240502 07:58:40.466089 24150 raft_consensus.cc:729] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f6079903ad93465c8741bf16583fe8c8, State: Initialized, Role: LEARNER
I20240502 07:58:40.466252 24150 consensus_queue.cc:260] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 175, Last appended: 1.175, Last appended by leader: 175, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 153 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: true } }
I20240502 07:58:40.466532 24150 ts_tablet_manager.cc:1398] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:40.467752 21823 tablet_copy_service.cc:342] P e8406d5a62054258be7fff420a4a4fe2: Request end of tablet copy session f6079903ad93465c8741bf16583fe8c8-efe71ca9e2604316b7b93385d4a9eace received from {username='slave'} at 127.18.13.133:57389
I20240502 07:58:40.467867 21823 tablet_copy_service.cc:433] P e8406d5a62054258be7fff420a4a4fe2: ending tablet copy session f6079903ad93465c8741bf16583fe8c8-efe71ca9e2604316b7b93385d4a9eace on tablet efe71ca9e2604316b7b93385d4a9eace with peer f6079903ad93465c8741bf16583fe8c8
I20240502 07:58:40.528271 23077 raft_consensus.cc:1203] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.152->[1.153-1.182] Dedup: 1.175->[1.176-1.182]
I20240502 07:58:40.584990 23672 raft_consensus.cc:1050] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2: attempting to promote NON_VOTER f6079903ad93465c8741bf16583fe8c8 to VOTER
I20240502 07:58:40.585407 23672 consensus_queue.cc:237] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 183, Committed index: 183, Last appended: 1.183, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } }
I20240502 07:58:40.587428 23075 raft_consensus.cc:1261] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 183. Preceding OpId from leader: term: 1 index: 184. (index mismatch)
I20240502 07:58:40.587987 23842 consensus_queue.cc:1035] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 184, Last known committed idx: 182, Time since last communication: 0.000s
I20240502 07:58:40.590444 23209 raft_consensus.cc:1261] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 183. Preceding OpId from leader: term: 1 index: 184. (index mismatch)
I20240502 07:58:40.593591 23672 consensus_queue.cc:1035] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 184, Last known committed idx: 183, Time since last communication: 0.000s
I20240502 07:58:40.598583 23672 raft_consensus.cc:2932] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.184: config changed from index 153 to 184, f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:40.599131 23209 raft_consensus.cc:2932] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Committing config change with OpId 1.184: config changed from index 153 to 184, f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:40.601979 21522 catalog_manager.cc:5484] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index 153 to 184, f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:40.620131 23075 raft_consensus.cc:2932] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [term 1 FOLLOWER]: Committing config change with OpId 1.184: config changed from index 153 to 184, f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:40.633471 23206 raft_consensus.cc:1203] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.168->[1.169-1.209] Dedup: 1.189->[1.190-1.209]
I20240502 07:58:40.639063 23076 raft_consensus.cc:1203] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.154->[1.155-1.173] Dedup: 1.163->[1.164-1.173]
I20240502 07:58:40.670248 23484 raft_consensus.cc:1050] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba: attempting to promote NON_VOTER c0ee0b72cadc4ebaa2e32a046a107ce6 to VOTER
I20240502 07:58:40.670676 23484 consensus_queue.cc:237] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 210, Committed index: 210, Last appended: 1.210, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } }
I20240502 07:58:40.672631 23208 raft_consensus.cc:1261] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 LEARNER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 210. Preceding OpId from leader: term: 1 index: 211. (index mismatch)
I20240502 07:58:40.672708 23340 raft_consensus.cc:1261] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 210. Preceding OpId from leader: term: 1 index: 211. (index mismatch)
I20240502 07:58:40.673076 24008 consensus_queue.cc:1035] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 211, Last known committed idx: 210, Time since last communication: 0.000s
I20240502 07:58:40.673239 24084 consensus_queue.cc:1035] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 211, Last known committed idx: 210, Time since last communication: 0.000s
I20240502 07:58:40.675242 22752 raft_consensus.cc:2932] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.211: config changed from index 169 to 211, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:40.675621 23208 raft_consensus.cc:2932] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Committing config change with OpId 1.211: config changed from index 169 to 211, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:40.681569 23340 raft_consensus.cc:2932] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Committing config change with OpId 1.211: config changed from index 169 to 211, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:40.682307 21529 catalog_manager.cc:5484] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba reported cstate change: config changed from index 169 to 211, c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:40.694564 24081 raft_consensus.cc:1050] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2: attempting to promote NON_VOTER f6079903ad93465c8741bf16583fe8c8 to VOTER
I20240502 07:58:40.694936 24081 consensus_queue.cc:237] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 174, Committed index: 174, Last appended: 1.174, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } }
I20240502 07:58:40.697247 23077 raft_consensus.cc:1261] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8 [term 1 LEARNER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 174. Preceding OpId from leader: term: 1 index: 175. (index mismatch)
I20240502 07:58:40.697291 23340 raft_consensus.cc:1261] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 174. Preceding OpId from leader: term: 1 index: 175. (index mismatch)
I20240502 07:58:40.697903 23673 consensus_queue.cc:1035] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 174, Time since last communication: 0.000s
I20240502 07:58:40.698326 23842 consensus_queue.cc:1035] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 175, Last known committed idx: 174, Time since last communication: 0.000s
I20240502 07:58:40.704792 24081 raft_consensus.cc:2932] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.175: config changed from index 155 to 175, f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:40.707623 23337 raft_consensus.cc:2932] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Committing config change with OpId 1.175: config changed from index 155 to 175, f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:40.708395 23077 raft_consensus.cc:2932] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8 [term 1 FOLLOWER]: Committing config change with OpId 1.175: config changed from index 155 to 175, f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:40.713224 21522 catalog_manager.cc:5484] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8 reported cstate change: config changed from index 155 to 175, f6079903ad93465c8741bf16583fe8c8 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:40.765347 21801 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e956783425bb49d2ab78be427e61781f"
dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55892
I20240502 07:58:40.765506 21801 raft_consensus.cc:595] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Received request to transfer leadership
W20240502 07:58:40.766551 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:40.772118 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:40.780371 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:40.789460 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:40.795236 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:40.804468 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:40.813998 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:40.823946 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:40.838676 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240502 07:58:40.841517 21803 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efe71ca9e2604316b7b93385d4a9eace"
dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55912
I20240502 07:58:40.841686 21803 raft_consensus.cc:595] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Received request to transfer leadership
W20240502 07:58:40.862115 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:40.879678 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:40.901252 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240502 07:58:40.902077 21931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "829291de88d84d6b8ead3a450430acba"
dest_uuid: "21b4d9bd384846b2ad74c199585018ba"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51148
I20240502 07:58:40.902274 21931 raft_consensus.cc:595] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Received request to transfer leadership
W20240502 07:58:40.912676 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:40.917430 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:40.920923 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:40.923152 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:40.923686 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:40.928304 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:40.929132 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:40.937680 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:40.939102 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:40.941716 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:40.946616 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:40.949568 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:40.955471 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:40.958766 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:40.958853 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:40.966416 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:40.970325 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:40.976442 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:40.976893 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:40.980666 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:40.985584 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:40.992430 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:40.998356 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:40.998967 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:41.006268 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:41.011430 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:41.018970 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:41.020061 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:41.023223 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:41.036113 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:41.037619 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:41.042490 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:41.052454 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:41.054139 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:41.066886 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:41.067907 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:41.072324 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:41.083041 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:41.091498 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:41.092970 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:41.102068 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:41.115312 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:41.119558 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:41.123889 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:41.134611 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:41.144152 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
I20240502 07:58:41.148659 21803 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e956783425bb49d2ab78be427e61781f"
dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55926
I20240502 07:58:41.148823 21803 raft_consensus.cc:595] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Received request to transfer leadership
W20240502 07:58:41.155051 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:41.157052 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:41.165786 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:41.178596 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
I20240502 07:58:41.181735 21803 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "efe71ca9e2604316b7b93385d4a9eace"
dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55948
I20240502 07:58:41.181891 21803 raft_consensus.cc:595] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Received request to transfer leadership
W20240502 07:58:41.187492 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:41.187503 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:41.201159 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
I20240502 07:58:41.205116 21931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "829291de88d84d6b8ead3a450430acba"
dest_uuid: "21b4d9bd384846b2ad74c199585018ba"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51166
I20240502 07:58:41.205276 21931 raft_consensus.cc:595] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Received request to transfer leadership
W20240502 07:58:41.212316 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:41.213061 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:41.228106 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:41.238723 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:41.243499 21761 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:41.253538 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
W20240502 07:58:41.266482 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
W20240502 07:58:41.272828 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
W20240502 07:58:41.278999 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Service unavailable: leader transfer in progress
I20240502 07:58:41.286078 24008 raft_consensus.cc:979] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba: : Instructing follower f97f8632bbe64638bb1ad2679849f434 to start an election
I20240502 07:58:41.286233 24008 raft_consensus.cc:1067] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Signalling peer f97f8632bbe64638bb1ad2679849f434 to start an election
I20240502 07:58:41.286861 23339 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "829291de88d84d6b8ead3a450430acba"
dest_uuid: "f97f8632bbe64638bb1ad2679849f434"
from {username='slave'} at 127.18.13.131:34427
I20240502 07:58:41.287178 23339 raft_consensus.cc:482] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:58:41.287335 23339 raft_consensus.cc:3037] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:41.289775 23339 raft_consensus.cc:504] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } }
I20240502 07:58:41.291164 23339 leader_election.cc:288] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [CANDIDATE]: Term 2 election: Requested vote from peers 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811), c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134:42113)
I20240502 07:58:41.291486 21931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829291de88d84d6b8ead3a450430acba" candidate_uuid: "f97f8632bbe64638bb1ad2679849f434" candidate_term: 2 candidate_status { last_received { term: 1 index: 215 } } ignore_live_leader: true dest_uuid: "21b4d9bd384846b2ad74c199585018ba"
I20240502 07:58:41.291710 21931 raft_consensus.cc:3032] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:58:41.292127 21931 raft_consensus.cc:729] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Running, Role: LEADER
I20240502 07:58:41.292377 21931 consensus_queue.cc:260] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 1.215, Last appended by leader: 215, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } }
I20240502 07:58:41.292618 21931 raft_consensus.cc:3037] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Advancing to term 2
W20240502 07:58:41.293428 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Service unavailable: leader transfer in progress
I20240502 07:58:41.299137 23206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829291de88d84d6b8ead3a450430acba" candidate_uuid: "f97f8632bbe64638bb1ad2679849f434" candidate_term: 2 candidate_status { last_received { term: 1 index: 215 } } ignore_live_leader: true dest_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6"
I20240502 07:58:41.299355 23206 raft_consensus.cc:3037] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:41.300595 23206 raft_consensus.cc:2454] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f97f8632bbe64638bb1ad2679849f434 in term 2.
I20240502 07:58:41.301103 23274 leader_election.cc:302] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ee0b72cadc4ebaa2e32a046a107ce6, f97f8632bbe64638bb1ad2679849f434; no voters:
I20240502 07:58:41.301558 24271 raft_consensus.cc:2781] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:41.301745 24271 raft_consensus.cc:686] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 2 LEADER]: Becoming Leader. State: Replica: f97f8632bbe64638bb1ad2679849f434, State: Running, Role: LEADER
I20240502 07:58:41.302103 24271 consensus_queue.cc:237] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 1.215, Last appended by leader: 215, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } }
W20240502 07:58:41.304489 21892 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50964: Illegal state: replica 21b4d9bd384846b2ad74c199585018ba is not leader of this config: current role FOLLOWER
W20240502 07:58:41.304571 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Service unavailable: leader transfer in progress
I20240502 07:58:41.294507 21931 raft_consensus.cc:2454] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f97f8632bbe64638bb1ad2679849f434 in term 2.
I20240502 07:58:41.304997 21526 catalog_manager.cc:5484] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 reported cstate change: term changed from 1 to 2, leader changed from 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) to f97f8632bbe64638bb1ad2679849f434 (127.18.13.135). New cstate: current_term: 2 leader_uuid: "f97f8632bbe64638bb1ad2679849f434" committed_config { opid_index: 211 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:41.313539 23673 raft_consensus.cc:979] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2: : Instructing follower f6079903ad93465c8741bf16583fe8c8 to start an election
I20240502 07:58:41.313694 23673 raft_consensus.cc:1067] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Signalling peer f6079903ad93465c8741bf16583fe8c8 to start an election
I20240502 07:58:41.314270 23076 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "efe71ca9e2604316b7b93385d4a9eace"
dest_uuid: "f6079903ad93465c8741bf16583fe8c8"
from {username='slave'} at 127.18.13.130:41641
I20240502 07:58:41.314507 23076 raft_consensus.cc:482] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:58:41.314637 23076 raft_consensus.cc:3037] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:41.315874 23076 raft_consensus.cc:504] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } }
I20240502 07:58:41.317158 23076 leader_election.cc:288] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [CANDIDATE]: Term 2 election: Requested vote from peers e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891), c0ee0b72cadc4ebaa2e32a046a107ce6 (127.18.13.134:42113)
I20240502 07:58:41.317633 21803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe71ca9e2604316b7b93385d4a9eace" candidate_uuid: "f6079903ad93465c8741bf16583fe8c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 189 } } ignore_live_leader: true dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
I20240502 07:58:41.317768 21803 raft_consensus.cc:3032] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:58:41.317813 21803 raft_consensus.cc:729] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Running, Role: LEADER
I20240502 07:58:41.317984 21803 consensus_queue.cc:260] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 189, Committed index: 189, Last appended: 1.189, Last appended by leader: 189, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } }
I20240502 07:58:41.318166 21803 raft_consensus.cc:3037] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:41.319360 21803 raft_consensus.cc:2454] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6079903ad93465c8741bf16583fe8c8 in term 2.
W20240502 07:58:41.320569 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55494: Illegal state: replica e8406d5a62054258be7fff420a4a4fe2 is not leader of this config: current role FOLLOWER
I20240502 07:58:41.324345 23012 leader_election.cc:302] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8406d5a62054258be7fff420a4a4fe2, f6079903ad93465c8741bf16583fe8c8; no voters:
I20240502 07:58:41.324985 23206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe71ca9e2604316b7b93385d4a9eace" candidate_uuid: "f6079903ad93465c8741bf16583fe8c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 189 } } ignore_live_leader: true dest_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6"
I20240502 07:58:41.325112 23206 raft_consensus.cc:3037] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:41.326185 23206 raft_consensus.cc:2454] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6079903ad93465c8741bf16583fe8c8 in term 2.
I20240502 07:58:41.328663 23842 raft_consensus.cc:979] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2: : Instructing follower f97f8632bbe64638bb1ad2679849f434 to start an election
I20240502 07:58:41.328778 23842 raft_consensus.cc:1067] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Signalling peer f97f8632bbe64638bb1ad2679849f434 to start an election
I20240502 07:58:41.329701 23339 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e956783425bb49d2ab78be427e61781f"
dest_uuid: "f97f8632bbe64638bb1ad2679849f434"
from {username='slave'} at 127.18.13.130:51443
I20240502 07:58:41.329859 23339 raft_consensus.cc:482] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:58:41.329932 23339 raft_consensus.cc:3037] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:41.335184 23339 raft_consensus.cc:504] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } }
I20240502 07:58:41.336269 21803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e956783425bb49d2ab78be427e61781f" candidate_uuid: "f97f8632bbe64638bb1ad2679849f434" candidate_term: 2 candidate_status { last_received { term: 1 index: 177 } } ignore_live_leader: true dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
I20240502 07:58:41.336395 21803 raft_consensus.cc:3032] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:58:41.336432 21803 raft_consensus.cc:729] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Running, Role: LEADER
I20240502 07:58:41.336568 21803 consensus_queue.cc:260] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 177, Committed index: 177, Last appended: 1.177, Last appended by leader: 177, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } }
W20240502 07:58:41.336650 21763 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Illegal state: replica e8406d5a62054258be7fff420a4a4fe2 is not leader of this config: current role FOLLOWER
I20240502 07:58:41.336701 21803 raft_consensus.cc:3037] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:41.337719 21803 raft_consensus.cc:2454] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f97f8632bbe64638bb1ad2679849f434 in term 2.
I20240502 07:58:41.337875 23339 leader_election.cc:288] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [CANDIDATE]: Term 2 election: Requested vote from peers e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891), f6079903ad93465c8741bf16583fe8c8 (127.18.13.133:38189)
I20240502 07:58:41.338130 23274 leader_election.cc:302] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8406d5a62054258be7fff420a4a4fe2, f97f8632bbe64638bb1ad2679849f434; no voters:
I20240502 07:58:41.338356 24272 raft_consensus.cc:2781] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:41.338440 24272 raft_consensus.cc:686] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 2 LEADER]: Becoming Leader. State: Replica: f97f8632bbe64638bb1ad2679849f434, State: Running, Role: LEADER
I20240502 07:58:41.338588 24272 consensus_queue.cc:237] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 177, Committed index: 177, Last appended: 1.177, Last appended by leader: 177, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } }
I20240502 07:58:41.341055 21526 catalog_manager.cc:5484] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 reported cstate change: term changed from 1 to 2, leader changed from e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) to f97f8632bbe64638bb1ad2679849f434 (127.18.13.135). New cstate: current_term: 2 leader_uuid: "f97f8632bbe64638bb1ad2679849f434" committed_config { opid_index: 175 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:41.347532 24277 raft_consensus.cc:2781] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:41.347777 24277 raft_consensus.cc:686] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [term 2 LEADER]: Becoming Leader. State: Replica: f6079903ad93465c8741bf16583fe8c8, State: Running, Role: LEADER
I20240502 07:58:41.348428 24277 consensus_queue.cc:237] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 189, Committed index: 189, Last appended: 1.189, Last appended by leader: 189, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } }
I20240502 07:58:41.351119 21526 catalog_manager.cc:5484] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 reported cstate change: term changed from 1 to 2, leader changed from e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) to f6079903ad93465c8741bf16583fe8c8 (127.18.13.133). New cstate: current_term: 2 leader_uuid: "f6079903ad93465c8741bf16583fe8c8" committed_config { opid_index: 184 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:41.351315 23076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e956783425bb49d2ab78be427e61781f" candidate_uuid: "f97f8632bbe64638bb1ad2679849f434" candidate_term: 2 candidate_status { last_received { term: 1 index: 177 } } ignore_live_leader: true dest_uuid: "f6079903ad93465c8741bf16583fe8c8"
I20240502 07:58:41.351493 23076 raft_consensus.cc:3037] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:41.352607 23076 raft_consensus.cc:2454] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f97f8632bbe64638bb1ad2679849f434 in term 2.
I20240502 07:58:41.400326 23206 raft_consensus.cc:1261] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 2 FOLLOWER]: Refusing update from remote peer f6079903ad93465c8741bf16583fe8c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 189. Preceding OpId from leader: term: 2 index: 191. (index mismatch)
I20240502 07:58:41.400872 24279 consensus_queue.cc:1035] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 190, Last known committed idx: 189, Time since last communication: 0.000s
I20240502 07:58:41.405186 21803 raft_consensus.cc:1261] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Refusing update from remote peer f6079903ad93465c8741bf16583fe8c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 189. Preceding OpId from leader: term: 2 index: 191. (index mismatch)
I20240502 07:58:41.409860 24195 mvcc.cc:205] Tried to move back new op lower bound from 7023152010844618752 to 7023152010646409216. Current Snapshot: MvccSnapshot[applied={T|T < 7023152010844618752 or (T in {7023152010844618752})}]
I20240502 07:58:41.411463 24277 consensus_queue.cc:1035] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 190, Last known committed idx: 189, Time since last communication: 0.000s
I20240502 07:58:41.416282 23209 raft_consensus.cc:1261] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 2 FOLLOWER]: Refusing update from remote peer f97f8632bbe64638bb1ad2679849f434: Log matching property violated. Preceding OpId in replica: term: 1 index: 215. Preceding OpId from leader: term: 2 index: 217. (index mismatch)
I20240502 07:58:41.417421 21934 raft_consensus.cc:1261] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 2 FOLLOWER]: Refusing update from remote peer f97f8632bbe64638bb1ad2679849f434: Log matching property violated. Preceding OpId in replica: term: 1 index: 215. Preceding OpId from leader: term: 2 index: 217. (index mismatch)
I20240502 07:58:41.417533 24271 consensus_queue.cc:1035] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20240502 07:58:41.418059 24271 consensus_queue.cc:1035] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s
I20240502 07:58:41.424525 23962 mvcc.cc:205] Tried to move back new op lower bound from 7023152010911711232 to 7023152010456731648. Current Snapshot: MvccSnapshot[applied={T|T < 7023152010911711232 or (T in {7023152010911711232})}]
I20240502 07:58:41.433053 21803 raft_consensus.cc:1261] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Refusing update from remote peer f97f8632bbe64638bb1ad2679849f434: Log matching property violated. Preceding OpId in replica: term: 1 index: 177. Preceding OpId from leader: term: 2 index: 179. (index mismatch)
I20240502 07:58:41.433552 23076 raft_consensus.cc:1261] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8 [term 2 FOLLOWER]: Refusing update from remote peer f97f8632bbe64638bb1ad2679849f434: Log matching property violated. Preceding OpId in replica: term: 1 index: 177. Preceding OpId from leader: term: 2 index: 179. (index mismatch)
I20240502 07:58:41.436097 24271 consensus_queue.cc:1035] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 178, Last known committed idx: 177, Time since last communication: 0.000s
I20240502 07:58:41.436436 24271 consensus_queue.cc:1035] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 178, Last known committed idx: 177, Time since last communication: 0.000s
I20240502 07:58:41.444695 23077 consensus_queue.cc:237] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 191, Committed index: 191, Last appended: 2.191, Last appended by leader: 189, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } }
I20240502 07:58:41.451274 23206 raft_consensus.cc:1261] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 2 FOLLOWER]: Refusing update from remote peer f6079903ad93465c8741bf16583fe8c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 191. Preceding OpId from leader: term: 2 index: 192. (index mismatch)
I20240502 07:58:41.452922 24277 consensus_queue.cc:1035] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 192, Last known committed idx: 191, Time since last communication: 0.000s
I20240502 07:58:41.458725 24279 raft_consensus.cc:2932] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 [term 2 LEADER]: Committing config change with OpId 2.192: config changed from index 184 to 192, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New config: { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:41.459861 23206 raft_consensus.cc:2932] T efe71ca9e2604316b7b93385d4a9eace P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 2 FOLLOWER]: Committing config change with OpId 2.192: config changed from index 184 to 192, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New config: { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:41.461251 21516 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet efe71ca9e2604316b7b93385d4a9eace with cas_config_opid_index 184: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:41.467962 21522 catalog_manager.cc:5484] T efe71ca9e2604316b7b93385d4a9eace P f6079903ad93465c8741bf16583fe8c8 reported cstate change: config changed from index 184 to 192, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New cstate: current_term: 2 leader_uuid: "f6079903ad93465c8741bf16583fe8c8" committed_config { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:41.476683 23339 consensus_queue.cc:237] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 219, Committed index: 219, Last appended: 2.219, Last appended by leader: 215, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } }
I20240502 07:58:41.482393 23339 consensus_queue.cc:237] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 180, Committed index: 180, Last appended: 2.180, Last appended by leader: 177, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } }
I20240502 07:58:41.483143 21782 tablet_service.cc:1514] Processing DeleteTablet for tablet efe71ca9e2604316b7b93385d4a9eace with delete_type TABLET_DATA_TOMBSTONED (TS e8406d5a62054258be7fff420a4a4fe2 not found in new config with opid_index 192) from {username='slave'} at 127.0.0.1:51132
I20240502 07:58:41.485397 23209 raft_consensus.cc:1261] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 2 FOLLOWER]: Refusing update from remote peer f97f8632bbe64638bb1ad2679849f434: Log matching property violated. Preceding OpId in replica: term: 2 index: 219. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20240502 07:58:41.486481 23077 raft_consensus.cc:1261] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8 [term 2 FOLLOWER]: Refusing update from remote peer f97f8632bbe64638bb1ad2679849f434: Log matching property violated. Preceding OpId in replica: term: 2 index: 180. Preceding OpId from leader: term: 2 index: 181. (index mismatch)
I20240502 07:58:41.487362 24298 consensus_queue.cc:1035] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I20240502 07:58:41.492635 24297 consensus_queue.cc:1035] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 220, Last known committed idx: 219, Time since last communication: 0.000s
I20240502 07:58:41.496081 24298 raft_consensus.cc:2932] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 [term 2 LEADER]: Committing config change with OpId 2.220: config changed from index 211 to 220, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:41.499960 24306 tablet_replica.cc:330] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2: stopping tablet replica
I20240502 07:58:41.500106 24306 raft_consensus.cc:2229] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:41.500209 24306 raft_consensus.cc:2258] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:41.500869 24306 ts_tablet_manager.cc:1875] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:41.500860 21514 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 829291de88d84d6b8ead3a450430acba with cas_config_opid_index 211: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:41.502864 21522 catalog_manager.cc:5484] T 829291de88d84d6b8ead3a450430acba P f97f8632bbe64638bb1ad2679849f434 reported cstate change: config changed from index 211 to 220, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New cstate: current_term: 2 leader_uuid: "f97f8632bbe64638bb1ad2679849f434" committed_config { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:41.503762 24306 ts_tablet_manager.cc:1888] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.191
I20240502 07:58:41.503867 24306 log.cc:1196] T efe71ca9e2604316b7b93385d4a9eace P e8406d5a62054258be7fff420a4a4fe2: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/efe71ca9e2604316b7b93385d4a9eace
I20240502 07:58:41.507683 21515 catalog_manager.cc:4830] TS e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891): tablet efe71ca9e2604316b7b93385d4a9eace (table rebalance_test_table_4 [id=83a6a35d6fa24f97bab7a6a64be5fe51]) successfully deleted
I20240502 07:58:41.508607 23206 raft_consensus.cc:2932] T 829291de88d84d6b8ead3a450430acba P c0ee0b72cadc4ebaa2e32a046a107ce6 [term 2 FOLLOWER]: Committing config change with OpId 2.220: config changed from index 211 to 220, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New config: { opid_index: 220 OBSOLETE_local: false peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c0ee0b72cadc4ebaa2e32a046a107ce6" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 42113 } attrs { promote: false } } }
I20240502 07:58:41.521045 21913 tablet_service.cc:1514] Processing DeleteTablet for tablet 829291de88d84d6b8ead3a450430acba with delete_type TABLET_DATA_TOMBSTONED (TS 21b4d9bd384846b2ad74c199585018ba not found in new config with opid_index 220) from {username='slave'} at 127.0.0.1:40950
I20240502 07:58:41.525820 24297 raft_consensus.cc:2932] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 [term 2 LEADER]: Committing config change with OpId 2.181: config changed from index 175 to 181, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:41.527961 24311 tablet_replica.cc:330] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba: stopping tablet replica
I20240502 07:58:41.528056 21514 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e956783425bb49d2ab78be427e61781f with cas_config_opid_index 175: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:41.528985 21526 catalog_manager.cc:5484] T e956783425bb49d2ab78be427e61781f P f97f8632bbe64638bb1ad2679849f434 reported cstate change: config changed from index 175 to 181, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New cstate: current_term: 2 leader_uuid: "f97f8632bbe64638bb1ad2679849f434" committed_config { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:41.529306 24311 raft_consensus.cc:2229] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:41.529498 24311 raft_consensus.cc:2258] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:41.530680 24311 ts_tablet_manager.cc:1875] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:41.534328 23076 raft_consensus.cc:2932] T e956783425bb49d2ab78be427e61781f P f6079903ad93465c8741bf16583fe8c8 [term 2 FOLLOWER]: Committing config change with OpId 2.181: config changed from index 175 to 181, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "f97f8632bbe64638bb1ad2679849f434" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "f6079903ad93465c8741bf16583fe8c8" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 38189 } attrs { promote: false } } }
I20240502 07:58:41.537484 24311 ts_tablet_manager.cc:1888] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.219
I20240502 07:58:41.539017 24311 log.cc:1196] T 829291de88d84d6b8ead3a450430acba P 21b4d9bd384846b2ad74c199585018ba: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/829291de88d84d6b8ead3a450430acba
I20240502 07:58:41.540127 21517 catalog_manager.cc:4830] TS 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811): tablet 829291de88d84d6b8ead3a450430acba (table rebalance_test_table_3 [id=aba27c7bd3a84e94b7e627dc4e8d5c6a]) successfully deleted
I20240502 07:58:41.543838 21782 tablet_service.cc:1514] Processing DeleteTablet for tablet e956783425bb49d2ab78be427e61781f with delete_type TABLET_DATA_TOMBSTONED (TS e8406d5a62054258be7fff420a4a4fe2 not found in new config with opid_index 181) from {username='slave'} at 127.0.0.1:51132
I20240502 07:58:41.544145 24306 tablet_replica.cc:330] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2: stopping tablet replica
I20240502 07:58:41.544310 24306 raft_consensus.cc:2229] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:41.544446 24306 raft_consensus.cc:2258] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:41.545557 24306 ts_tablet_manager.cc:1875] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:41.549157 24306 ts_tablet_manager.cc:1888] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.180
I20240502 07:58:41.551882 24306 log.cc:1196] T e956783425bb49d2ab78be427e61781f P e8406d5a62054258be7fff420a4a4fe2: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/e956783425bb49d2ab78be427e61781f
I20240502 07:58:41.557545 21515 catalog_manager.cc:4830] TS e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891): tablet e956783425bb49d2ab78be427e61781f (table rebalance_test_table_2 [id=4f8126ed45de428eaf540e5d65bd2c86]) successfully deleted
I20240502 07:58:41.877801 23189 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:41.882802 21913 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:41.887221 23058 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:41.935537 23319 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:41.955132 21782 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:41.977634 21651 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:41.985051 22927 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:42.271109 21913 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:42.277029 23189 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:42.280879 21651 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:42.288381 22927 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:42.300011 21782 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:42.315088 23319 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:42.316130 23058 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:42.946877 21651 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:42.974509 23319 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:42.979010 21913 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:42.991250 23189 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:42.994987 23058 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:43.000406 21782 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:43.019040 22927 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:43.412282 21913 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:43.412922 21651 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:43.419849 21782 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:43.424037 22927 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:43.428898 23189 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:43.466614 23319 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:43.476461 23058 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:43.749580 22927 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:43.750562 21913 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:43.756161 21782 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:43.757265 23189 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:43.760792 23319 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:43.765358 21651 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:43.765697 23058 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
146dd163130c4e40ab0cc570b6cc2f0e | 127.18.13.190:33713 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.13.148:45155 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
21b4d9bd384846b2ad74c199585018ba | 127.18.13.131:38811 | HEALTHY | <none> | 3 | 0
c0ee0b72cadc4ebaa2e32a046a107ce6 | 127.18.13.134:42113 | HEALTHY | <none> | 1 | 0
e218451f770e4ab0a5266e60bdb30a6a | 127.18.13.129:40667 | HEALTHY | <none> | 4 | 0
e8406d5a62054258be7fff420a4a4fe2 | 127.18.13.130:40891 | HEALTHY | <none> | 4 | 0
ec7796d40c8a463caa442aa67ba3d367 | 127.18.13.132:40025 | HEALTHY | <none> | 1 | 0
f6079903ad93465c8741bf16583fe8c8 | 127.18.13.133:38189 | HEALTHY | <none> | 1 | 0
f97f8632bbe64638bb1ad2679849f434 | 127.18.13.135:43407 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.18.13.129 | experimental | 127.18.13.129:40667
local_ip_for_outbound_sockets | 127.18.13.130 | experimental | 127.18.13.130:40891
local_ip_for_outbound_sockets | 127.18.13.131 | experimental | 127.18.13.131:38811
local_ip_for_outbound_sockets | 127.18.13.132 | experimental | 127.18.13.132:40025
local_ip_for_outbound_sockets | 127.18.13.133 | experimental | 127.18.13.133:38189
local_ip_for_outbound_sockets | 127.18.13.134 | experimental | 127.18.13.134:42113
local_ip_for_outbound_sockets | 127.18.13.135 | experimental | 127.18.13.135:43407
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.13.129:40667
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.13.130:40891
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.13.131:38811
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.18.13.132:40025
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.13.133:38189
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.13.134:42113
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.18.13.135:43407
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.13.148:45155 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 07:58:43.803689 18486 log_verifier.cc:126] Checking tablet 0029538492ea4f25a822f18de9c83c54
I20240502 07:58:43.861855 18486 log_verifier.cc:177] Verified matching terms for 294 ops in tablet 0029538492ea4f25a822f18de9c83c54
I20240502 07:58:43.861982 18486 log_verifier.cc:126] Checking tablet 01e44d65c0314f0a9d30a2587723c290
I20240502 07:58:43.877053 18486 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 01e44d65c0314f0a9d30a2587723c290
I20240502 07:58:43.877126 18486 log_verifier.cc:126] Checking tablet 10c7c018e7994c71a77f6d547e0c6bb5
I20240502 07:58:43.934616 18486 log_verifier.cc:177] Verified matching terms for 296 ops in tablet 10c7c018e7994c71a77f6d547e0c6bb5
I20240502 07:58:43.934772 18486 log_verifier.cc:126] Checking tablet 1968b639a6f54f8eb31f90d2ad0c0748
I20240502 07:58:43.988637 18486 log_verifier.cc:177] Verified matching terms for 230 ops in tablet 1968b639a6f54f8eb31f90d2ad0c0748
I20240502 07:58:43.988782 18486 log_verifier.cc:126] Checking tablet 2ed04eb8c46f4cf4bf721d75baa8d534
I20240502 07:58:44.040961 18486 log_verifier.cc:177] Verified matching terms for 255 ops in tablet 2ed04eb8c46f4cf4bf721d75baa8d534
I20240502 07:58:44.041079 18486 log_verifier.cc:126] Checking tablet 4b138b61a1604cd18fe31b0f4ffb28d2
I20240502 07:58:44.098189 18486 log_verifier.cc:177] Verified matching terms for 292 ops in tablet 4b138b61a1604cd18fe31b0f4ffb28d2
I20240502 07:58:44.098325 18486 log_verifier.cc:126] Checking tablet 56f307e7a3f34ed18e4a6f5272034279
I20240502 07:58:44.149736 18486 log_verifier.cc:177] Verified matching terms for 219 ops in tablet 56f307e7a3f34ed18e4a6f5272034279
I20240502 07:58:44.149897 18486 log_verifier.cc:126] Checking tablet 829291de88d84d6b8ead3a450430acba
I20240502 07:58:44.211750 18486 log_verifier.cc:177] Verified matching terms for 293 ops in tablet 829291de88d84d6b8ead3a450430acba
I20240502 07:58:44.211966 18486 log_verifier.cc:126] Checking tablet c725ad05375344c3b52682bc91090a43
I20240502 07:58:44.266229 18486 log_verifier.cc:177] Verified matching terms for 277 ops in tablet c725ad05375344c3b52682bc91090a43
I20240502 07:58:44.266353 18486 log_verifier.cc:126] Checking tablet d59002c98d3f4ab8bbea19caaf0c8419
I20240502 07:58:44.315585 18486 log_verifier.cc:177] Verified matching terms for 235 ops in tablet d59002c98d3f4ab8bbea19caaf0c8419
I20240502 07:58:44.315706 18486 log_verifier.cc:126] Checking tablet e0c289e06aba4976ab7249d72f26c0e0
I20240502 07:58:44.372606 18486 log_verifier.cc:177] Verified matching terms for 302 ops in tablet e0c289e06aba4976ab7249d72f26c0e0
I20240502 07:58:44.372745 18486 log_verifier.cc:126] Checking tablet e46a300254bb45d1970dfb5db271ad14
I20240502 07:58:44.429777 18486 log_verifier.cc:177] Verified matching terms for 299 ops in tablet e46a300254bb45d1970dfb5db271ad14
I20240502 07:58:44.429934 18486 log_verifier.cc:126] Checking tablet e956783425bb49d2ab78be427e61781f
I20240502 07:58:44.481885 18486 log_verifier.cc:177] Verified matching terms for 237 ops in tablet e956783425bb49d2ab78be427e61781f
I20240502 07:58:44.482035 18486 log_verifier.cc:126] Checking tablet efe71ca9e2604316b7b93385d4a9eace
I20240502 07:58:44.538144 18486 log_verifier.cc:177] Verified matching terms for 264 ops in tablet efe71ca9e2604316b7b93385d4a9eace
I20240502 07:58:44.538297 18486 log_verifier.cc:126] Checking tablet f3f58339fd2c4528a2decb910ea73e21
I20240502 07:58:44.593276 18486 log_verifier.cc:177] Verified matching terms for 270 ops in tablet f3f58339fd2c4528a2decb910ea73e21
I20240502 07:58:44.593428 18486 log_verifier.cc:126] Checking tablet fd14aec4cc3f4b8fa6e3d0bb1241a240
I20240502 07:58:44.644060 18486 log_verifier.cc:177] Verified matching terms for 225 ops in tablet fd14aec4cc3f4b8fa6e3d0bb1241a240
I20240502 07:58:44.646416 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.136:0
--local_ip_for_outbound_sockets=127.18.13.136
--webserver_interface=127.18.13.136
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--builtin_ntp_servers=127.18.13.148:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 07:58:44.765784 24536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:44.766047 24536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:44.766127 24536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:44.769636 24536 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:44.769734 24536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:44.769855 24536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.136
I20240502 07:58:44.773924 24536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.18.13.136
--webserver_port=0
--tserver_master_addrs=127.18.13.190:33713
--never_fsync=true
--heap_profile_path=/tmp/kudu.24536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.136
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:44.774982 24536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:44.776274 24536 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:44.786608 24541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:44.787709 24542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:44.788586 24544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:44.789669 24536 server_base.cc:1034] running on GCE node
I20240502 07:58:44.790127 24536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:44.790779 24536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:44.791929 24536 hybrid_clock.cc:648] HybridClock initialized: now 1714636724791906 us; error 40 us; skew 500 ppm
I20240502 07:58:44.794162 24536 webserver.cc:460] Webserver started at http://127.18.13.136:39977/ using document root <none> and password file <none>
I20240502 07:58:44.794785 24536 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:44.794848 24536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:44.795058 24536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:44.796929 24536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "1ff74b20491b47bcb48bf5b0ac2e290c"
format_stamp: "Formatted at 2024-05-02 07:58:44 on dist-test-slave-f46z"
I20240502 07:58:44.797430 24536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "1ff74b20491b47bcb48bf5b0ac2e290c"
format_stamp: "Formatted at 2024-05-02 07:58:44 on dist-test-slave-f46z"
I20240502 07:58:44.801115 24536 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240502 07:58:44.803594 24550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:44.804808 24536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:44.804934 24536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "1ff74b20491b47bcb48bf5b0ac2e290c"
format_stamp: "Formatted at 2024-05-02 07:58:44 on dist-test-slave-f46z"
I20240502 07:58:44.805055 24536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:44.818558 24536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:44.819310 24536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:44.819499 24536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:44.820178 24536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:44.821130 24536 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:44.821178 24536 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:44.821228 24536 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:44.821266 24536 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:44.831387 24536 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.136:40477
I20240502 07:58:44.831409 24663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.136:40477 every 8 connection(s)
I20240502 07:58:44.832463 24536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 07:58:44.833415 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 24536
I20240502 07:58:44.833537 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 07:58:44.843530 24664 heartbeater.cc:347] Connected to a master server at 127.18.13.190:33713
I20240502 07:58:44.843825 24664 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:44.844455 24664 heartbeater.cc:511] Master 127.18.13.190:33713 requested a full tablet report, sending...
I20240502 07:58:44.845656 21530 ts_manager.cc:194] Registered new tserver with Master: 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136:40477)
I20240502 07:58:44.846375 21530 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.136:53895
I20240502 07:58:45.045341 23189 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:45.047870 21913 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:45.048166 22927 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:45.048838 21782 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:45.063171 23058 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:45.068414 21651 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:45.083987 24587 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:45.085417 23319 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:45.178663 24587 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:45.179450 21913 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:45.182859 21782 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:45.184525 23189 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:45.184973 21651 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:45.197928 23058 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:45.206050 22927 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:45.220155 23319 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:45.252686 21931 consensus_queue.cc:237] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 1.296, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } }
I20240502 07:58:45.254346 21803 raft_consensus.cc:1261] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 1 index: 297. (index mismatch)
I20240502 07:58:45.254772 24759 consensus_queue.cc:1035] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s
I20240502 07:58:45.256680 23484 raft_consensus.cc:2932] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.297: config changed from index -1 to 297, NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) added. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } } }
I20240502 07:58:45.257894 21803 raft_consensus.cc:2932] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Committing config change with OpId 1.297: config changed from index -1 to 297, NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) added. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } } }
W20240502 07:58:45.258621 21867 consensus_peers.cc:485] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba -> Peer 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136:40477): Couldn't send request to peer 1ff74b20491b47bcb48bf5b0ac2e290c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10c7c018e7994c71a77f6d547e0c6bb5. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:45.259598 21529 catalog_manager.cc:5484] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba reported cstate change: config changed from index -1 to 297, NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:45.268321 21668 consensus_queue.cc:237] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 225, Committed index: 225, Last appended: 1.225, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } }
I20240502 07:58:45.270107 22946 raft_consensus.cc:1261] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 225. Preceding OpId from leader: term: 1 index: 226. (index mismatch)
I20240502 07:58:45.270536 24772 consensus_queue.cc:1035] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 226, Last known committed idx: 225, Time since last communication: 0.000s
I20240502 07:58:45.272382 22677 raft_consensus.cc:2932] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.226: config changed from index 57 to 226, NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } } }
I20240502 07:58:45.272934 22946 raft_consensus.cc:2932] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Committing config change with OpId 1.226: config changed from index 57 to 226, NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) added. New config: { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } } }
W20240502 07:58:45.273177 21605 consensus_peers.cc:485] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a -> Peer 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136:40477): Couldn't send request to peer 1ff74b20491b47bcb48bf5b0ac2e290c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd14aec4cc3f4b8fa6e3d0bb1241a240. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:45.275260 21530 catalog_manager.cc:5484] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: config changed from index 57 to 226, NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:45.285792 21803 consensus_queue.cc:237] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 1.235, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } }
I20240502 07:58:45.287715 22946 raft_consensus.cc:1261] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 1 index: 236. (index mismatch)
I20240502 07:58:45.288199 24781 consensus_queue.cc:1035] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s
I20240502 07:58:45.290953 24781 raft_consensus.cc:2932] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.236: config changed from index 66 to 236, NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } } }
I20240502 07:58:45.291597 22946 raft_consensus.cc:2932] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Committing config change with OpId 1.236: config changed from index 66 to 236, NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) added. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } } }
W20240502 07:58:45.292066 21736 consensus_peers.cc:485] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 -> Peer 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136:40477): Couldn't send request to peer 1ff74b20491b47bcb48bf5b0ac2e290c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d59002c98d3f4ab8bbea19caaf0c8419. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:45.293968 21529 catalog_manager.cc:5484] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index 66 to 236, NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:45.302608 21931 consensus_queue.cc:237] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } }
I20240502 07:58:45.304137 22946 raft_consensus.cc:1261] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 07:58:45.304234 21867 consensus_peers.cc:485] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba -> Peer 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136:40477): Couldn't send request to peer 1ff74b20491b47bcb48bf5b0ac2e290c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01e44d65c0314f0a9d30a2587723c290. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:45.304528 24759 consensus_queue.cc:1035] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:58:45.306432 23484 raft_consensus.cc:2932] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } } }
I20240502 07:58:45.306869 22946 raft_consensus.cc:2932] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } } }
I20240502 07:58:45.309104 21530 catalog_manager.cc:5484] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba reported cstate change: config changed from index 4 to 5, NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:45.746387 24833 ts_tablet_manager.cc:902] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c: Initiating tablet copy from peer e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891)
I20240502 07:58:45.747267 24833 tablet_copy_client.cc:308] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:40891
I20240502 07:58:45.751035 21823 tablet_copy_service.cc:140] P e8406d5a62054258be7fff420a4a4fe2: Received BeginTabletCopySession request for tablet d59002c98d3f4ab8bbea19caaf0c8419 from peer 1ff74b20491b47bcb48bf5b0ac2e290c ({username='slave'} at 127.18.13.136:36047)
I20240502 07:58:45.751173 21823 tablet_copy_service.cc:161] P e8406d5a62054258be7fff420a4a4fe2: Beginning new tablet copy session on tablet d59002c98d3f4ab8bbea19caaf0c8419 from peer 1ff74b20491b47bcb48bf5b0ac2e290c at {username='slave'} at 127.18.13.136:36047: session id = 1ff74b20491b47bcb48bf5b0ac2e290c-d59002c98d3f4ab8bbea19caaf0c8419
I20240502 07:58:45.752131 21823 tablet_copy_source_session.cc:205] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:45.755318 24833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d59002c98d3f4ab8bbea19caaf0c8419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:45.760591 24833 tablet_copy_client.cc:783] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:45.760802 24833 tablet_copy_client.cc:647] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:45.763047 24833 tablet_copy_client.cc:523] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:45.764856 24833 tablet_bootstrap.cc:492] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c: Bootstrap starting.
I20240502 07:58:45.788766 24833 log.cc:824] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:45.848676 24664 heartbeater.cc:503] Master 127.18.13.190:33713 was elected leader, sending a full tablet report...
I20240502 07:58:45.849905 24860 ts_tablet_manager.cc:902] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c: Initiating tablet copy from peer 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:45.850509 24860 tablet_copy_client.cc:308] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:38811
I20240502 07:58:45.854426 21954 tablet_copy_service.cc:140] P 21b4d9bd384846b2ad74c199585018ba: Received BeginTabletCopySession request for tablet 01e44d65c0314f0a9d30a2587723c290 from peer 1ff74b20491b47bcb48bf5b0ac2e290c ({username='slave'} at 127.18.13.136:45471)
I20240502 07:58:45.854560 21954 tablet_copy_service.cc:161] P 21b4d9bd384846b2ad74c199585018ba: Beginning new tablet copy session on tablet 01e44d65c0314f0a9d30a2587723c290 from peer 1ff74b20491b47bcb48bf5b0ac2e290c at {username='slave'} at 127.18.13.136:45471: session id = 1ff74b20491b47bcb48bf5b0ac2e290c-01e44d65c0314f0a9d30a2587723c290
I20240502 07:58:45.855309 21954 tablet_copy_source_session.cc:205] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:45.856177 24860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e44d65c0314f0a9d30a2587723c290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:45.858549 24860 tablet_copy_client.cc:783] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:45.858752 24860 tablet_copy_client.cc:647] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:45.860107 24860 tablet_copy_client.cc:523] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:45.861580 24860 tablet_bootstrap.cc:492] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c: Bootstrap starting.
I20240502 07:58:45.866398 24860 tablet_bootstrap.cc:492] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:45.866812 24860 tablet_bootstrap.cc:492] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c: Bootstrap complete.
I20240502 07:58:45.867164 24860 ts_tablet_manager.cc:1367] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 07:58:45.870410 24860 raft_consensus.cc:348] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } }
I20240502 07:58:45.870738 24860 raft_consensus.cc:729] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ff74b20491b47bcb48bf5b0ac2e290c, State: Initialized, Role: LEARNER
I20240502 07:58:45.871147 24860 consensus_queue.cc:260] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } }
I20240502 07:58:45.871584 24866 ts_tablet_manager.cc:902] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c: Initiating tablet copy from peer 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811)
I20240502 07:58:45.872193 24866 tablet_copy_client.cc:308] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:38811
I20240502 07:58:45.873209 21954 tablet_copy_service.cc:140] P 21b4d9bd384846b2ad74c199585018ba: Received BeginTabletCopySession request for tablet 10c7c018e7994c71a77f6d547e0c6bb5 from peer 1ff74b20491b47bcb48bf5b0ac2e290c ({username='slave'} at 127.18.13.136:45471)
I20240502 07:58:45.873317 21954 tablet_copy_service.cc:161] P 21b4d9bd384846b2ad74c199585018ba: Beginning new tablet copy session on tablet 10c7c018e7994c71a77f6d547e0c6bb5 from peer 1ff74b20491b47bcb48bf5b0ac2e290c at {username='slave'} at 127.18.13.136:45471: session id = 1ff74b20491b47bcb48bf5b0ac2e290c-10c7c018e7994c71a77f6d547e0c6bb5
I20240502 07:58:45.872299 24860 ts_tablet_manager.cc:1398] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20240502 07:58:45.874226 21954 tablet_copy_source_session.cc:205] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:45.875201 24866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c7c018e7994c71a77f6d547e0c6bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:45.875767 21954 tablet_copy_service.cc:342] P 21b4d9bd384846b2ad74c199585018ba: Request end of tablet copy session 1ff74b20491b47bcb48bf5b0ac2e290c-01e44d65c0314f0a9d30a2587723c290 received from {username='slave'} at 127.18.13.136:45471
I20240502 07:58:45.875869 21954 tablet_copy_service.cc:433] P 21b4d9bd384846b2ad74c199585018ba: ending tablet copy session 1ff74b20491b47bcb48bf5b0ac2e290c-01e44d65c0314f0a9d30a2587723c290 on tablet 01e44d65c0314f0a9d30a2587723c290 with peer 1ff74b20491b47bcb48bf5b0ac2e290c
I20240502 07:58:45.878098 24866 tablet_copy_client.cc:783] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:45.878254 24866 tablet_copy_client.cc:647] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:45.880507 24866 tablet_copy_client.cc:523] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:45.881739 24866 tablet_bootstrap.cc:492] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c: Bootstrap starting.
I20240502 07:58:45.894203 24860 ts_tablet_manager.cc:902] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c: Initiating tablet copy from peer e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667)
I20240502 07:58:45.894830 24860 tablet_copy_client.cc:308] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:40667
I20240502 07:58:45.899521 21692 tablet_copy_service.cc:140] P e218451f770e4ab0a5266e60bdb30a6a: Received BeginTabletCopySession request for tablet fd14aec4cc3f4b8fa6e3d0bb1241a240 from peer 1ff74b20491b47bcb48bf5b0ac2e290c ({username='slave'} at 127.18.13.136:46985)
I20240502 07:58:45.899664 21692 tablet_copy_service.cc:161] P e218451f770e4ab0a5266e60bdb30a6a: Beginning new tablet copy session on tablet fd14aec4cc3f4b8fa6e3d0bb1241a240 from peer 1ff74b20491b47bcb48bf5b0ac2e290c at {username='slave'} at 127.18.13.136:46985: session id = 1ff74b20491b47bcb48bf5b0ac2e290c-fd14aec4cc3f4b8fa6e3d0bb1241a240
I20240502 07:58:45.900578 21692 tablet_copy_source_session.cc:205] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:45.901499 24860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd14aec4cc3f4b8fa6e3d0bb1241a240. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:45.903733 24860 tablet_copy_client.cc:783] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:45.904292 24860 tablet_copy_client.cc:647] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:45.905875 24860 tablet_copy_client.cc:523] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:45.907511 24860 tablet_bootstrap.cc:492] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c: Bootstrap starting.
I20240502 07:58:45.936388 24833 tablet_bootstrap.cc:492] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c: Bootstrap replayed 1/1 log segments. Stats: ops{read=236 overwritten=0 applied=236 ignored=0} inserts{seen=864 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:45.936707 24833 tablet_bootstrap.cc:492] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c: Bootstrap complete.
I20240502 07:58:45.936935 24833 ts_tablet_manager.cc:1367] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c: Time spent bootstrapping tablet: real 0.172s user 0.156s sys 0.015s
I20240502 07:58:45.937582 24833 raft_consensus.cc:348] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } }
I20240502 07:58:45.937719 24833 raft_consensus.cc:729] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ff74b20491b47bcb48bf5b0ac2e290c, State: Initialized, Role: LEARNER
I20240502 07:58:45.937834 24833 consensus_queue.cc:260] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 236, Last appended: 1.236, Last appended by leader: 236, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } }
I20240502 07:58:45.938127 24833 ts_tablet_manager.cc:1398] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:45.938925 21823 tablet_copy_service.cc:342] P e8406d5a62054258be7fff420a4a4fe2: Request end of tablet copy session 1ff74b20491b47bcb48bf5b0ac2e290c-d59002c98d3f4ab8bbea19caaf0c8419 received from {username='slave'} at 127.18.13.136:36047
I20240502 07:58:45.939021 21823 tablet_copy_service.cc:433] P e8406d5a62054258be7fff420a4a4fe2: ending tablet copy session 1ff74b20491b47bcb48bf5b0ac2e290c-d59002c98d3f4ab8bbea19caaf0c8419 on tablet d59002c98d3f4ab8bbea19caaf0c8419 with peer 1ff74b20491b47bcb48bf5b0ac2e290c
I20240502 07:58:46.100775 24860 tablet_bootstrap.cc:492] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c: Bootstrap replayed 1/1 log segments. Stats: ops{read=226 overwritten=0 applied=226 ignored=0} inserts{seen=908 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:46.101161 24860 tablet_bootstrap.cc:492] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c: Bootstrap complete.
I20240502 07:58:46.101406 24860 ts_tablet_manager.cc:1367] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c: Time spent bootstrapping tablet: real 0.194s user 0.176s sys 0.012s
I20240502 07:58:46.101997 24860 raft_consensus.cc:348] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } }
I20240502 07:58:46.102149 24860 raft_consensus.cc:729] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ff74b20491b47bcb48bf5b0ac2e290c, State: Initialized, Role: LEARNER
I20240502 07:58:46.102277 24860 consensus_queue.cc:260] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 226, Last appended: 1.226, Last appended by leader: 226, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 226 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } }
I20240502 07:58:46.102619 24860 ts_tablet_manager.cc:1398] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:46.103366 21692 tablet_copy_service.cc:342] P e218451f770e4ab0a5266e60bdb30a6a: Request end of tablet copy session 1ff74b20491b47bcb48bf5b0ac2e290c-fd14aec4cc3f4b8fa6e3d0bb1241a240 received from {username='slave'} at 127.18.13.136:46985
I20240502 07:58:46.103474 21692 tablet_copy_service.cc:433] P e218451f770e4ab0a5266e60bdb30a6a: ending tablet copy session 1ff74b20491b47bcb48bf5b0ac2e290c-fd14aec4cc3f4b8fa6e3d0bb1241a240 on tablet fd14aec4cc3f4b8fa6e3d0bb1241a240 with peer 1ff74b20491b47bcb48bf5b0ac2e290c
I20240502 07:58:46.106192 24866 tablet_bootstrap.cc:492] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c: Bootstrap replayed 1/1 log segments. Stats: ops{read=297 overwritten=0 applied=297 ignored=0} inserts{seen=928 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:46.106549 24866 tablet_bootstrap.cc:492] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c: Bootstrap complete.
I20240502 07:58:46.106777 24866 ts_tablet_manager.cc:1367] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c: Time spent bootstrapping tablet: real 0.225s user 0.208s sys 0.012s
I20240502 07:58:46.107273 24866 raft_consensus.cc:348] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } }
I20240502 07:58:46.107410 24866 raft_consensus.cc:729] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1ff74b20491b47bcb48bf5b0ac2e290c, State: Initialized, Role: LEARNER
I20240502 07:58:46.107517 24866 consensus_queue.cc:260] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 297, Last appended: 1.297, Last appended by leader: 297, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: true } }
I20240502 07:58:46.108019 24866 ts_tablet_manager.cc:1398] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 07:58:46.108544 21954 tablet_copy_service.cc:342] P 21b4d9bd384846b2ad74c199585018ba: Request end of tablet copy session 1ff74b20491b47bcb48bf5b0ac2e290c-10c7c018e7994c71a77f6d547e0c6bb5 received from {username='slave'} at 127.18.13.136:45471
I20240502 07:58:46.108647 21954 tablet_copy_service.cc:433] P 21b4d9bd384846b2ad74c199585018ba: ending tablet copy session 1ff74b20491b47bcb48bf5b0ac2e290c-10c7c018e7994c71a77f6d547e0c6bb5 on tablet 10c7c018e7994c71a77f6d547e0c6bb5 with peer 1ff74b20491b47bcb48bf5b0ac2e290c
I20240502 07:58:46.236614 24605 raft_consensus.cc:1203] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:58:46.314185 24605 raft_consensus.cc:1203] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.296->[1.297-1.297] Dedup: 1.297->[]
I20240502 07:58:46.326388 24605 raft_consensus.cc:1203] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.225->[1.226-1.226] Dedup: 1.226->[]
I20240502 07:58:46.360073 24605 raft_consensus.cc:1203] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.235->[1.236-1.236] Dedup: 1.236->[]
I20240502 07:58:46.747886 24925 raft_consensus.cc:1050] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2: attempting to promote NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c to VOTER
I20240502 07:58:46.748312 24925 consensus_queue.cc:237] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 236, Committed index: 236, Last appended: 1.236, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:46.748914 24906 raft_consensus.cc:1050] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba: attempting to promote NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c to VOTER
I20240502 07:58:46.749364 24906 consensus_queue.cc:237] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 1.297, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:46.749761 22946 raft_consensus.cc:1261] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 1 index: 237. (index mismatch)
I20240502 07:58:46.750417 24784 consensus_queue.cc:1035] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20240502 07:58:46.750751 21803 raft_consensus.cc:1261] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 297. Preceding OpId from leader: term: 1 index: 298. (index mismatch)
I20240502 07:58:46.750870 24609 raft_consensus.cc:1261] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 297. Preceding OpId from leader: term: 1 index: 298. (index mismatch)
I20240502 07:58:46.751055 24605 raft_consensus.cc:1261] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 1 index: 237. (index mismatch)
I20240502 07:58:46.751303 24759 consensus_queue.cc:1035] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I20240502 07:58:46.751636 24759 consensus_queue.cc:1035] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s
I20240502 07:58:46.752368 24925 consensus_queue.cc:1035] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 237, Last known committed idx: 236, Time since last communication: 0.000s
I20240502 07:58:46.753093 24925 raft_consensus.cc:2932] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Committing config change with OpId 1.237: config changed from index 236 to 237, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:46.754370 24906 raft_consensus.cc:2932] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.298: config changed from index 297 to 298, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:46.756319 21530 catalog_manager.cc:5484] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index 236 to 237, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:46.756305 22946 raft_consensus.cc:2932] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Committing config change with OpId 1.237: config changed from index 236 to 237, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:46.756500 21803 raft_consensus.cc:2932] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Committing config change with OpId 1.298: config changed from index 297 to 298, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:46.757551 24609 raft_consensus.cc:2932] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 FOLLOWER]: Committing config change with OpId 1.237: config changed from index 236 to 237, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:46.758388 24605 raft_consensus.cc:2932] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 FOLLOWER]: Committing config change with OpId 1.298: config changed from index 297 to 298, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:46.759038 21529 catalog_manager.cc:5484] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba reported cstate change: config changed from index 297 to 298, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:46.807329 24759 raft_consensus.cc:1050] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba: attempting to promote NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c to VOTER
I20240502 07:58:46.807821 24759 consensus_queue.cc:237] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:46.809324 24605 raft_consensus.cc:1261] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:58:46.809423 22946 raft_consensus.cc:1261] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:58:46.809809 24759 consensus_queue.cc:1035] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:58:46.810128 24905 consensus_queue.cc:1035] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:58:46.812239 24906 raft_consensus.cc:2932] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:46.812805 22946 raft_consensus.cc:2932] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:46.812805 24605 raft_consensus.cc:2932] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:46.815681 21523 catalog_manager.cc:5484] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367 reported cstate change: config changed from index 5 to 6, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:46.818936 21931 consensus_queue.cc:237] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:46.820123 24605 raft_consensus.cc:1261] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 FOLLOWER]: Refusing update from remote peer 21b4d9bd384846b2ad74c199585018ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:58:46.820516 24905 consensus_queue.cc:1035] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:58:46.822136 24906 raft_consensus.cc:2932] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:46.822484 24605 raft_consensus.cc:2932] T 01e44d65c0314f0a9d30a2587723c290 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:46.823671 21517 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 01e44d65c0314f0a9d30a2587723c290 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:46.825105 21526 catalog_manager.cc:5484] T 01e44d65c0314f0a9d30a2587723c290 P 21b4d9bd384846b2ad74c199585018ba reported cstate change: config changed from index 6 to 7, VOTER ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132) evicted. New cstate: current_term: 1 leader_uuid: "21b4d9bd384846b2ad74c199585018ba" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:46.827744 22927 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e44d65c0314f0a9d30a2587723c290 with delete_type TABLET_DATA_TOMBSTONED (TS ec7796d40c8a463caa442aa67ba3d367 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59072
I20240502 07:58:46.828449 24983 tablet_replica.cc:330] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367: stopping tablet replica
I20240502 07:58:46.828806 24983 raft_consensus.cc:2229] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:46.829073 24983 raft_consensus.cc:2258] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:46.830786 24983 ts_tablet_manager.cc:1875] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:46.833626 24983 ts_tablet_manager.cc:1888] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 07:58:46.833727 24983 log.cc:1196] T 01e44d65c0314f0a9d30a2587723c290 P ec7796d40c8a463caa442aa67ba3d367: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/wals/01e44d65c0314f0a9d30a2587723c290
I20240502 07:58:46.834602 21514 catalog_manager.cc:4830] TS ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132:40025): tablet 01e44d65c0314f0a9d30a2587723c290 (table TestTable [id=828c706eb9dd489bb081a01b553ce7ff]) successfully deleted
I20240502 07:58:46.914511 21803 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d59002c98d3f4ab8bbea19caaf0c8419"
dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39640
I20240502 07:58:46.914665 21803 raft_consensus.cc:595] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:58:46.925751 21931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10c7c018e7994c71a77f6d547e0c6bb5"
dest_uuid: "21b4d9bd384846b2ad74c199585018ba"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38406
I20240502 07:58:46.925904 21931 raft_consensus.cc:595] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Received request to transfer leadership
I20240502 07:58:46.937721 24923 raft_consensus.cc:1050] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a: attempting to promote NON_VOTER 1ff74b20491b47bcb48bf5b0ac2e290c to VOTER
I20240502 07:58:46.938155 24923 consensus_queue.cc:237] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 226, Committed index: 226, Last appended: 1.226, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:46.939347 24605 raft_consensus.cc:1261] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 LEARNER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 226. Preceding OpId from leader: term: 1 index: 227. (index mismatch)
I20240502 07:58:46.939383 22946 raft_consensus.cc:1261] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Refusing update from remote peer e218451f770e4ab0a5266e60bdb30a6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 226. Preceding OpId from leader: term: 1 index: 227. (index mismatch)
I20240502 07:58:46.939765 25017 consensus_queue.cc:1035] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 227, Last known committed idx: 226, Time since last communication: 0.000s
I20240502 07:58:46.940044 25015 consensus_queue.cc:1035] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 227, Last known committed idx: 226, Time since last communication: 0.000s
I20240502 07:58:46.942440 25017 raft_consensus.cc:2932] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Committing config change with OpId 1.227: config changed from index 226 to 227, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:46.942550 24605 raft_consensus.cc:2932] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 FOLLOWER]: Committing config change with OpId 1.227: config changed from index 226 to 227, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:46.944593 22946 raft_consensus.cc:2932] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Committing config change with OpId 1.227: config changed from index 226 to 227, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:46.945277 21530 catalog_manager.cc:5484] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a reported cstate change: config changed from index 226 to 227, 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e218451f770e4ab0a5266e60bdb30a6a" committed_config { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:46.953555 21803 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d59002c98d3f4ab8bbea19caaf0c8419"
dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39644
I20240502 07:58:46.953707 21803 raft_consensus.cc:595] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:58:46.964579 21931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10c7c018e7994c71a77f6d547e0c6bb5"
dest_uuid: "21b4d9bd384846b2ad74c199585018ba"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38438
I20240502 07:58:46.964731 21931 raft_consensus.cc:595] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Received request to transfer leadership
I20240502 07:58:46.976209 21668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fd14aec4cc3f4b8fa6e3d0bb1241a240"
dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41348
I20240502 07:58:46.976361 21668 raft_consensus.cc:595] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Received request to transfer leadership
I20240502 07:58:47.187557 21803 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d59002c98d3f4ab8bbea19caaf0c8419"
dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39670
I20240502 07:58:47.187718 21803 raft_consensus.cc:595] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:58:47.198565 21931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10c7c018e7994c71a77f6d547e0c6bb5"
dest_uuid: "21b4d9bd384846b2ad74c199585018ba"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38458
I20240502 07:58:47.198714 21931 raft_consensus.cc:595] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Received request to transfer leadership
I20240502 07:58:47.209621 21668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fd14aec4cc3f4b8fa6e3d0bb1241a240"
dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41374
I20240502 07:58:47.209777 21668 raft_consensus.cc:595] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Received request to transfer leadership
I20240502 07:58:47.268734 24925 raft_consensus.cc:979] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2: : Instructing follower 1ff74b20491b47bcb48bf5b0ac2e290c to start an election
I20240502 07:58:47.268854 24925 raft_consensus.cc:1067] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Signalling peer 1ff74b20491b47bcb48bf5b0ac2e290c to start an election
I20240502 07:58:47.269373 24605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d59002c98d3f4ab8bbea19caaf0c8419"
dest_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c"
from {username='slave'} at 127.18.13.130:39185
I20240502 07:58:47.269608 24605 raft_consensus.cc:482] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:58:47.269726 24605 raft_consensus.cc:3037] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:47.270713 24605 raft_consensus.cc:504] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:47.271739 24605 leader_election.cc:288] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [CANDIDATE]: Term 2 election: Requested vote from peers e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891), ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132:40025)
I20240502 07:58:47.272198 21803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59002c98d3f4ab8bbea19caaf0c8419" candidate_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" candidate_term: 2 candidate_status { last_received { term: 1 index: 237 } } ignore_live_leader: true dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
I20240502 07:58:47.272356 21803 raft_consensus.cc:3032] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:58:47.272409 21803 raft_consensus.cc:729] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Running, Role: LEADER
I20240502 07:58:47.272579 21803 consensus_queue.cc:260] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 1.237, Last appended by leader: 237, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:47.272771 21803 raft_consensus.cc:3037] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:47.273836 21803 raft_consensus.cc:2454] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff74b20491b47bcb48bf5b0ac2e290c in term 2.
I20240502 07:58:47.274240 24552 leader_election.cc:302] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff74b20491b47bcb48bf5b0ac2e290c, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:47.274464 24867 raft_consensus.cc:2781] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:47.274679 24867 raft_consensus.cc:686] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 2 LEADER]: Becoming Leader. State: Replica: 1ff74b20491b47bcb48bf5b0ac2e290c, State: Running, Role: LEADER
I20240502 07:58:47.275000 24867 consensus_queue.cc:237] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 1.237, Last appended by leader: 237, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:47.276944 22946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d59002c98d3f4ab8bbea19caaf0c8419" candidate_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" candidate_term: 2 candidate_status { last_received { term: 1 index: 237 } } ignore_live_leader: true dest_uuid: "ec7796d40c8a463caa442aa67ba3d367"
I20240502 07:58:47.277144 22946 raft_consensus.cc:3037] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:47.277647 21523 catalog_manager.cc:5484] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c reported cstate change: term changed from 1 to 2, leader changed from e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) to 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136). New cstate: current_term: 2 leader_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" committed_config { opid_index: 237 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:47.278292 22946 raft_consensus.cc:2454] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff74b20491b47bcb48bf5b0ac2e290c in term 2.
I20240502 07:58:47.352646 24905 raft_consensus.cc:979] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba: : Instructing follower e8406d5a62054258be7fff420a4a4fe2 to start an election
I20240502 07:58:47.352771 24905 raft_consensus.cc:1067] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Signalling peer e8406d5a62054258be7fff420a4a4fe2 to start an election
I20240502 07:58:47.353278 21803 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10c7c018e7994c71a77f6d547e0c6bb5"
dest_uuid: "e8406d5a62054258be7fff420a4a4fe2"
from {username='slave'} at 127.18.13.131:46645
I20240502 07:58:47.353457 21803 raft_consensus.cc:482] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:58:47.353521 21803 raft_consensus.cc:3037] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:47.354427 21803 raft_consensus.cc:504] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:47.354882 21803 leader_election.cc:288] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 2 election: Requested vote from peers 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811), 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136:40477)
I20240502 07:58:47.355250 21931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c7c018e7994c71a77f6d547e0c6bb5" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 2 candidate_status { last_received { term: 1 index: 298 } } ignore_live_leader: true dest_uuid: "21b4d9bd384846b2ad74c199585018ba"
I20240502 07:58:47.355386 21931 raft_consensus.cc:3032] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:58:47.355448 21931 raft_consensus.cc:729] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 21b4d9bd384846b2ad74c199585018ba, State: Running, Role: LEADER
I20240502 07:58:47.355437 24605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c7c018e7994c71a77f6d547e0c6bb5" candidate_uuid: "e8406d5a62054258be7fff420a4a4fe2" candidate_term: 2 candidate_status { last_received { term: 1 index: 298 } } ignore_live_leader: true dest_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c"
I20240502 07:58:47.355649 24605 raft_consensus.cc:3037] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:47.355648 21931 consensus_queue.cc:260] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 298, Committed index: 298, Last appended: 1.298, Last appended by leader: 298, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:47.355885 21931 raft_consensus.cc:3037] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:47.356523 24605 raft_consensus.cc:2454] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 2.
I20240502 07:58:47.356738 21931 raft_consensus.cc:2454] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8406d5a62054258be7fff420a4a4fe2 in term 2.
I20240502 07:58:47.356882 21736 leader_election.cc:302] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff74b20491b47bcb48bf5b0ac2e290c, e8406d5a62054258be7fff420a4a4fe2; no voters:
I20240502 07:58:47.357061 24925 raft_consensus.cc:2781] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:47.357143 24925 raft_consensus.cc:686] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 2 LEADER]: Becoming Leader. State: Replica: e8406d5a62054258be7fff420a4a4fe2, State: Running, Role: LEADER
I20240502 07:58:47.357298 24925 consensus_queue.cc:237] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 298, Committed index: 298, Last appended: 1.298, Last appended by leader: 298, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:47.358893 21523 catalog_manager.cc:5484] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: term changed from 1 to 2, leader changed from 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) to e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130). New cstate: current_term: 2 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:47.432719 21668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fd14aec4cc3f4b8fa6e3d0bb1241a240"
dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41384
I20240502 07:58:47.432863 21668 raft_consensus.cc:595] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Received request to transfer leadership
I20240502 07:58:47.444512 25106 raft_consensus.cc:979] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a: : Instructing follower ec7796d40c8a463caa442aa67ba3d367 to start an election
I20240502 07:58:47.444684 25106 raft_consensus.cc:1067] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Signalling peer ec7796d40c8a463caa442aa67ba3d367 to start an election
I20240502 07:58:47.445120 22946 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fd14aec4cc3f4b8fa6e3d0bb1241a240"
dest_uuid: "ec7796d40c8a463caa442aa67ba3d367"
from {username='slave'} at 127.18.13.129:57753
I20240502 07:58:47.445302 22946 raft_consensus.cc:482] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:58:47.445365 22946 raft_consensus.cc:3037] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:47.446246 22946 raft_consensus.cc:504] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:47.446727 22946 leader_election.cc:288] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [CANDIDATE]: Term 2 election: Requested vote from peers e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667), 1ff74b20491b47bcb48bf5b0ac2e290c (127.18.13.136:40477)
I20240502 07:58:47.447085 21668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd14aec4cc3f4b8fa6e3d0bb1241a240" candidate_uuid: "ec7796d40c8a463caa442aa67ba3d367" candidate_term: 2 candidate_status { last_received { term: 1 index: 227 } } ignore_live_leader: true dest_uuid: "e218451f770e4ab0a5266e60bdb30a6a"
I20240502 07:58:47.447242 21668 raft_consensus.cc:3032] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:58:47.447284 21668 raft_consensus.cc:729] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e218451f770e4ab0a5266e60bdb30a6a, State: Running, Role: LEADER
I20240502 07:58:47.447424 21668 consensus_queue.cc:260] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 1.227, Last appended by leader: 227, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:47.447630 21668 raft_consensus.cc:3037] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:47.448700 21668 raft_consensus.cc:2454] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec7796d40c8a463caa442aa67ba3d367 in term 2.
I20240502 07:58:47.449041 22881 leader_election.cc:302] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e218451f770e4ab0a5266e60bdb30a6a, ec7796d40c8a463caa442aa67ba3d367; no voters:
I20240502 07:58:47.449203 24968 raft_consensus.cc:2781] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:47.449415 24968 raft_consensus.cc:686] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 2 LEADER]: Becoming Leader. State: Replica: ec7796d40c8a463caa442aa67ba3d367, State: Running, Role: LEADER
I20240502 07:58:47.449580 24968 consensus_queue.cc:237] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 1.227, Last appended by leader: 227, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:47.450809 24605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd14aec4cc3f4b8fa6e3d0bb1241a240" candidate_uuid: "ec7796d40c8a463caa442aa67ba3d367" candidate_term: 2 candidate_status { last_received { term: 1 index: 227 } } ignore_live_leader: true dest_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c"
I20240502 07:58:47.450935 24605 raft_consensus.cc:3037] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:47.451422 21523 catalog_manager.cc:5484] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 reported cstate change: term changed from 1 to 2, leader changed from e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) to ec7796d40c8a463caa442aa67ba3d367 (127.18.13.132). New cstate: current_term: 2 leader_uuid: "ec7796d40c8a463caa442aa67ba3d367" committed_config { opid_index: 227 OBSOLETE_local: false peers { permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:47.452008 24605 raft_consensus.cc:2454] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec7796d40c8a463caa442aa67ba3d367 in term 2.
I20240502 07:58:47.821120 21931 raft_consensus.cc:1261] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 2 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 298. Preceding OpId from leader: term: 2 index: 299. (index mismatch)
I20240502 07:58:47.821627 24925 consensus_queue.cc:1035] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b4d9bd384846b2ad74c199585018ba" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38811 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 299, Last known committed idx: 298, Time since last communication: 0.000s
I20240502 07:58:47.824832 24605 raft_consensus.cc:1261] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 2 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 1 index: 298. Preceding OpId from leader: term: 2 index: 299. (index mismatch)
I20240502 07:58:47.825225 24925 consensus_queue.cc:1035] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 299, Last known committed idx: 298, Time since last communication: 0.000s
I20240502 07:58:47.831655 21803 raft_consensus.cc:1261] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Refusing update from remote peer 1ff74b20491b47bcb48bf5b0ac2e290c: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 2 index: 238. (index mismatch)
I20240502 07:58:47.832090 25128 consensus_queue.cc:1035] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20240502 07:58:47.834913 22946 raft_consensus.cc:1261] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 2 FOLLOWER]: Refusing update from remote peer 1ff74b20491b47bcb48bf5b0ac2e290c: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 2 index: 238. (index mismatch)
I20240502 07:58:47.835475 25128 consensus_queue.cc:1035] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.000s
I20240502 07:58:47.862994 21668 raft_consensus.cc:1261] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 2 FOLLOWER]: Refusing update from remote peer ec7796d40c8a463caa442aa67ba3d367: Log matching property violated. Preceding OpId in replica: term: 1 index: 227. Preceding OpId from leader: term: 2 index: 228. (index mismatch)
I20240502 07:58:47.863438 24968 consensus_queue.cc:1035] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218451f770e4ab0a5266e60bdb30a6a" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40667 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20240502 07:58:47.866138 24605 raft_consensus.cc:1261] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 2 FOLLOWER]: Refusing update from remote peer ec7796d40c8a463caa442aa67ba3d367: Log matching property violated. Preceding OpId in replica: term: 1 index: 227. Preceding OpId from leader: term: 2 index: 228. (index mismatch)
I20240502 07:58:47.866621 25145 consensus_queue.cc:1035] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s
I20240502 07:58:48.305284 24605 consensus_queue.cc:237] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 2.238, Last appended by leader: 237, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:48.306555 22946 raft_consensus.cc:1261] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 2 FOLLOWER]: Refusing update from remote peer 1ff74b20491b47bcb48bf5b0ac2e290c: Log matching property violated. Preceding OpId in replica: term: 2 index: 238. Preceding OpId from leader: term: 2 index: 239. (index mismatch)
I20240502 07:58:48.306938 25179 consensus_queue.cc:1035] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240502 07:58:48.308790 25132 raft_consensus.cc:2932] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 2 LEADER]: Committing config change with OpId 2.239: config changed from index 237 to 239, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:48.309250 22947 raft_consensus.cc:2932] T d59002c98d3f4ab8bbea19caaf0c8419 P ec7796d40c8a463caa442aa67ba3d367 [term 2 FOLLOWER]: Committing config change with OpId 2.239: config changed from index 237 to 239, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:48.310233 21514 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d59002c98d3f4ab8bbea19caaf0c8419 with cas_config_opid_index 237: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:48.311694 21526 catalog_manager.cc:5484] T d59002c98d3f4ab8bbea19caaf0c8419 P 1ff74b20491b47bcb48bf5b0ac2e290c reported cstate change: config changed from index 237 to 239, VOTER e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130) evicted. New cstate: current_term: 2 leader_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" committed_config { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:48.314502 21782 tablet_service.cc:1514] Processing DeleteTablet for tablet d59002c98d3f4ab8bbea19caaf0c8419 with delete_type TABLET_DATA_TOMBSTONED (TS e8406d5a62054258be7fff420a4a4fe2 not found in new config with opid_index 239) from {username='slave'} at 127.0.0.1:51132
I20240502 07:58:48.314937 25189 tablet_replica.cc:330] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2: stopping tablet replica
I20240502 07:58:48.315052 25189 raft_consensus.cc:2229] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:48.315157 25189 raft_consensus.cc:2258] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:48.316535 25189 ts_tablet_manager.cc:1875] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:48.318406 25189 ts_tablet_manager.cc:1888] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.238
I20240502 07:58:48.318473 25189 log.cc:1196] T d59002c98d3f4ab8bbea19caaf0c8419 P e8406d5a62054258be7fff420a4a4fe2: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/d59002c98d3f4ab8bbea19caaf0c8419
I20240502 07:58:48.319036 21515 catalog_manager.cc:4830] TS e8406d5a62054258be7fff420a4a4fe2 (127.18.13.130:40891): tablet d59002c98d3f4ab8bbea19caaf0c8419 (table rebalance_test_table_4 [id=83a6a35d6fa24f97bab7a6a64be5fe51]) successfully deleted
I20240502 07:58:48.321013 21803 consensus_queue.cc:237] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 299, Committed index: 299, Last appended: 2.299, Last appended by leader: 298, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:48.322021 24605 raft_consensus.cc:1261] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 2 FOLLOWER]: Refusing update from remote peer e8406d5a62054258be7fff420a4a4fe2: Log matching property violated. Preceding OpId in replica: term: 2 index: 299. Preceding OpId from leader: term: 2 index: 300. (index mismatch)
I20240502 07:58:48.322446 25190 consensus_queue.cc:1035] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 300, Last known committed idx: 299, Time since last communication: 0.000s
I20240502 07:58:48.324055 25080 raft_consensus.cc:2932] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 [term 2 LEADER]: Committing config change with OpId 2.300: config changed from index 298 to 300, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:48.324483 24605 raft_consensus.cc:2932] T 10c7c018e7994c71a77f6d547e0c6bb5 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 2 FOLLOWER]: Committing config change with OpId 2.300: config changed from index 298 to 300, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:48.326167 21515 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 10c7c018e7994c71a77f6d547e0c6bb5 with cas_config_opid_index 298: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:48.326869 21523 catalog_manager.cc:5484] T 10c7c018e7994c71a77f6d547e0c6bb5 P e8406d5a62054258be7fff420a4a4fe2 reported cstate change: config changed from index 298 to 300, VOTER 21b4d9bd384846b2ad74c199585018ba (127.18.13.131) evicted. New cstate: current_term: 2 leader_uuid: "e8406d5a62054258be7fff420a4a4fe2" committed_config { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "e8406d5a62054258be7fff420a4a4fe2" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:48.329193 21913 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c7c018e7994c71a77f6d547e0c6bb5 with delete_type TABLET_DATA_TOMBSTONED (TS 21b4d9bd384846b2ad74c199585018ba not found in new config with opid_index 300) from {username='slave'} at 127.0.0.1:40950
I20240502 07:58:48.329877 25191 tablet_replica.cc:330] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba: stopping tablet replica
I20240502 07:58:48.330014 25191 raft_consensus.cc:2229] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:48.330132 25191 raft_consensus.cc:2258] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:48.331564 25191 ts_tablet_manager.cc:1875] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:48.333550 25191 ts_tablet_manager.cc:1888] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.299
I20240502 07:58:48.333628 25191 log.cc:1196] T 10c7c018e7994c71a77f6d547e0c6bb5 P 21b4d9bd384846b2ad74c199585018ba: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/10c7c018e7994c71a77f6d547e0c6bb5
I20240502 07:58:48.334220 21517 catalog_manager.cc:4830] TS 21b4d9bd384846b2ad74c199585018ba (127.18.13.131:38811): tablet 10c7c018e7994c71a77f6d547e0c6bb5 (table rebalance_test_table_3 [id=aba27c7bd3a84e94b7e627dc4e8d5c6a]) successfully deleted
I20240502 07:58:48.487620 22946 consensus_queue.cc:237] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 228, Committed index: 228, Last appended: 2.228, Last appended by leader: 227, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } }
I20240502 07:58:48.488827 24605 raft_consensus.cc:1261] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 2 FOLLOWER]: Refusing update from remote peer ec7796d40c8a463caa442aa67ba3d367: Log matching property violated. Preceding OpId in replica: term: 2 index: 228. Preceding OpId from leader: term: 2 index: 229. (index mismatch)
I20240502 07:58:48.489234 25193 consensus_queue.cc:1035] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 229, Last known committed idx: 228, Time since last communication: 0.000s
I20240502 07:58:48.490850 25145 raft_consensus.cc:2932] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 [term 2 LEADER]: Committing config change with OpId 2.229: config changed from index 227 to 229, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:48.491941 24605 raft_consensus.cc:2932] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P 1ff74b20491b47bcb48bf5b0ac2e290c [term 2 FOLLOWER]: Committing config change with OpId 2.229: config changed from index 227 to 229, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New config: { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } } }
I20240502 07:58:48.492533 21514 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fd14aec4cc3f4b8fa6e3d0bb1241a240 with cas_config_opid_index 227: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:48.493181 21526 catalog_manager.cc:5484] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P ec7796d40c8a463caa442aa67ba3d367 reported cstate change: config changed from index 227 to 229, VOTER e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129) evicted. New cstate: current_term: 2 leader_uuid: "ec7796d40c8a463caa442aa67ba3d367" committed_config { opid_index: 229 OBSOLETE_local: false peers { permanent_uuid: "ec7796d40c8a463caa442aa67ba3d367" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 40025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff74b20491b47bcb48bf5b0ac2e290c" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 40477 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:48.495612 21651 tablet_service.cc:1514] Processing DeleteTablet for tablet fd14aec4cc3f4b8fa6e3d0bb1241a240 with delete_type TABLET_DATA_TOMBSTONED (TS e218451f770e4ab0a5266e60bdb30a6a not found in new config with opid_index 229) from {username='slave'} at 127.0.0.1:46142
I20240502 07:58:48.496021 25194 tablet_replica.cc:330] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a: stopping tablet replica
I20240502 07:58:48.496191 25194 raft_consensus.cc:2229] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:48.496323 25194 raft_consensus.cc:2258] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:48.498075 25194 ts_tablet_manager.cc:1875] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:48.500528 25194 ts_tablet_manager.cc:1888] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.228
I20240502 07:58:48.500605 25194 log.cc:1196] T fd14aec4cc3f4b8fa6e3d0bb1241a240 P e218451f770e4ab0a5266e60bdb30a6a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/fd14aec4cc3f4b8fa6e3d0bb1241a240
I20240502 07:58:48.501164 21515 catalog_manager.cc:4830] TS e218451f770e4ab0a5266e60bdb30a6a (127.18.13.129:40667): tablet fd14aec4cc3f4b8fa6e3d0bb1241a240 (table rebalance_test_table_2 [id=4f8126ed45de428eaf540e5d65bd2c86]) successfully deleted
I20240502 07:58:48.591701 21913 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:48.591864 22927 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:48.592682 23189 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:48.594458 24587 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:48.595702 21651 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:48.606961 21782 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:48.626670 23058 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:48.630936 23319 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:48.716702 21651 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:48.718041 23189 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:48.718626 24587 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:48.722184 21782 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:48.722314 21913 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:48.740460 22927 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:48.754282 23058 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:48.755126 23319 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:48.855526 21913 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:48.860937 21782 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:48.862085 22927 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:48.862505 21651 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:48.863818 23189 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:48.866456 23319 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:58:48.872488 24587 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:48.876410 23058 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
146dd163130c4e40ab0cc570b6cc2f0e | 127.18.13.190:33713 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.13.148:45155 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
1ff74b20491b47bcb48bf5b0ac2e290c | 127.18.13.136:40477 | HEALTHY | <none> | 1 | 0
21b4d9bd384846b2ad74c199585018ba | 127.18.13.131:38811 | HEALTHY | <none> | 2 | 0
c0ee0b72cadc4ebaa2e32a046a107ce6 | 127.18.13.134:42113 | HEALTHY | <none> | 1 | 0
e218451f770e4ab0a5266e60bdb30a6a | 127.18.13.129:40667 | HEALTHY | <none> | 3 | 0
e8406d5a62054258be7fff420a4a4fe2 | 127.18.13.130:40891 | HEALTHY | <none> | 4 | 0
ec7796d40c8a463caa442aa67ba3d367 | 127.18.13.132:40025 | HEALTHY | <none> | 2 | 0
f6079903ad93465c8741bf16583fe8c8 | 127.18.13.133:38189 | HEALTHY | <none> | 1 | 0
f97f8632bbe64638bb1ad2679849f434 | 127.18.13.135:43407 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.18.13.129 | experimental | 127.18.13.129:40667
local_ip_for_outbound_sockets | 127.18.13.130 | experimental | 127.18.13.130:40891
local_ip_for_outbound_sockets | 127.18.13.131 | experimental | 127.18.13.131:38811
local_ip_for_outbound_sockets | 127.18.13.132 | experimental | 127.18.13.132:40025
local_ip_for_outbound_sockets | 127.18.13.133 | experimental | 127.18.13.133:38189
local_ip_for_outbound_sockets | 127.18.13.134 | experimental | 127.18.13.134:42113
local_ip_for_outbound_sockets | 127.18.13.135 | experimental | 127.18.13.135:43407
local_ip_for_outbound_sockets | 127.18.13.136 | experimental | 127.18.13.136:40477
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.13.129:40667
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.13.130:40891
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.13.131:38811
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.18.13.132:40025
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.18.13.133:38189
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.18.13.134:42113
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.18.13.135:43407
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.18.13.136:40477
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.13.148:45155 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 07:58:48.926038 18486 log_verifier.cc:126] Checking tablet 0029538492ea4f25a822f18de9c83c54
I20240502 07:58:48.978075 18486 log_verifier.cc:177] Verified matching terms for 294 ops in tablet 0029538492ea4f25a822f18de9c83c54
I20240502 07:58:48.978206 18486 log_verifier.cc:126] Checking tablet 01e44d65c0314f0a9d30a2587723c290
I20240502 07:58:48.990891 18486 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 01e44d65c0314f0a9d30a2587723c290
I20240502 07:58:48.990964 18486 log_verifier.cc:126] Checking tablet 10c7c018e7994c71a77f6d547e0c6bb5
I20240502 07:58:49.045701 18486 log_verifier.cc:177] Verified matching terms for 300 ops in tablet 10c7c018e7994c71a77f6d547e0c6bb5
I20240502 07:58:49.045902 18486 log_verifier.cc:126] Checking tablet 1968b639a6f54f8eb31f90d2ad0c0748
I20240502 07:58:49.088552 18486 log_verifier.cc:177] Verified matching terms for 230 ops in tablet 1968b639a6f54f8eb31f90d2ad0c0748
I20240502 07:58:49.088689 18486 log_verifier.cc:126] Checking tablet 2ed04eb8c46f4cf4bf721d75baa8d534
I20240502 07:58:49.137758 18486 log_verifier.cc:177] Verified matching terms for 255 ops in tablet 2ed04eb8c46f4cf4bf721d75baa8d534
I20240502 07:58:49.137913 18486 log_verifier.cc:126] Checking tablet 4b138b61a1604cd18fe31b0f4ffb28d2
I20240502 07:58:49.196230 18486 log_verifier.cc:177] Verified matching terms for 292 ops in tablet 4b138b61a1604cd18fe31b0f4ffb28d2
I20240502 07:58:49.196388 18486 log_verifier.cc:126] Checking tablet 56f307e7a3f34ed18e4a6f5272034279
I20240502 07:58:49.243822 18486 log_verifier.cc:177] Verified matching terms for 219 ops in tablet 56f307e7a3f34ed18e4a6f5272034279
I20240502 07:58:49.243966 18486 log_verifier.cc:126] Checking tablet 829291de88d84d6b8ead3a450430acba
I20240502 07:58:49.297407 18486 log_verifier.cc:177] Verified matching terms for 293 ops in tablet 829291de88d84d6b8ead3a450430acba
I20240502 07:58:49.297566 18486 log_verifier.cc:126] Checking tablet c725ad05375344c3b52682bc91090a43
I20240502 07:58:49.348340 18486 log_verifier.cc:177] Verified matching terms for 277 ops in tablet c725ad05375344c3b52682bc91090a43
I20240502 07:58:49.348485 18486 log_verifier.cc:126] Checking tablet d59002c98d3f4ab8bbea19caaf0c8419
I20240502 07:58:49.394905 18486 log_verifier.cc:177] Verified matching terms for 239 ops in tablet d59002c98d3f4ab8bbea19caaf0c8419
I20240502 07:58:49.395056 18486 log_verifier.cc:126] Checking tablet e0c289e06aba4976ab7249d72f26c0e0
I20240502 07:58:49.452324 18486 log_verifier.cc:177] Verified matching terms for 302 ops in tablet e0c289e06aba4976ab7249d72f26c0e0
I20240502 07:58:49.452483 18486 log_verifier.cc:126] Checking tablet e46a300254bb45d1970dfb5db271ad14
I20240502 07:58:49.504971 18486 log_verifier.cc:177] Verified matching terms for 299 ops in tablet e46a300254bb45d1970dfb5db271ad14
I20240502 07:58:49.505121 18486 log_verifier.cc:126] Checking tablet e956783425bb49d2ab78be427e61781f
I20240502 07:58:49.551132 18486 log_verifier.cc:177] Verified matching terms for 237 ops in tablet e956783425bb49d2ab78be427e61781f
I20240502 07:58:49.551273 18486 log_verifier.cc:126] Checking tablet efe71ca9e2604316b7b93385d4a9eace
I20240502 07:58:49.599485 18486 log_verifier.cc:177] Verified matching terms for 264 ops in tablet efe71ca9e2604316b7b93385d4a9eace
I20240502 07:58:49.599653 18486 log_verifier.cc:126] Checking tablet f3f58339fd2c4528a2decb910ea73e21
I20240502 07:58:49.647779 18486 log_verifier.cc:177] Verified matching terms for 270 ops in tablet f3f58339fd2c4528a2decb910ea73e21
I20240502 07:58:49.647972 18486 log_verifier.cc:126] Checking tablet fd14aec4cc3f4b8fa6e3d0bb1241a240
I20240502 07:58:49.693759 18486 log_verifier.cc:177] Verified matching terms for 229 ops in tablet fd14aec4cc3f4b8fa6e3d0bb1241a240
I20240502 07:58:49.695416 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 21590
I20240502 07:58:49.704384 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 21721
I20240502 07:58:49.711589 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 21852
I20240502 07:58:49.718897 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 22865
I20240502 07:58:49.725183 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 22996
I20240502 07:58:49.731299 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 23127
I20240502 07:58:49.737797 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 23258
I20240502 07:58:49.743744 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 24536
I20240502 07:58:49.749883 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 21499
2024-05-02T07:58:49Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/3 (25114 ms)
[----------] 1 test from RebalanceParamTest (25114 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240502 07:58:49.806336 18486 test_util.cc:268] Using random seed: -114102790
I20240502 07:58:49.807149 18486 ts_itest-base.cc:112] Starting cluster with:
I20240502 07:58:49.807204 18486 ts_itest-base.cc:113] --------------
I20240502 07:58:49.807215 18486 ts_itest-base.cc:114] 8 tablet servers
I20240502 07:58:49.807225 18486 ts_itest-base.cc:115] 3 replicas per TS
I20240502 07:58:49.807235 18486 ts_itest-base.cc:116] --------------
2024-05-02T07:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T07:58:49Z Disabled control of system clock
I20240502 07:58:49.816164 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.13.190:46035
--webserver_interface=127.18.13.190
--webserver_port=0
--builtin_ntp_servers=127.18.13.148:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.13.190:46035
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 07:58:49.927668 25325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:49.928011 25325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:49.928092 25325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:49.931245 25325 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:49.931346 25325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 07:58:49.931387 25325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:49.931432 25325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 07:58:49.931473 25325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 07:58:49.935372 25325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.18.13.190:46035
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.13.190:46035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.18.13.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.25325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:49.936456 25325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:49.937672 25325 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:49.945753 25331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:49.946791 25333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:49.947809 25330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:49.949280 25325 server_base.cc:1034] running on GCE node
I20240502 07:58:49.949862 25325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:49.950874 25325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:49.952032 25325 hybrid_clock.cc:648] HybridClock initialized: now 1714636729952015 us; error 43 us; skew 500 ppm
I20240502 07:58:49.954133 25325 webserver.cc:460] Webserver started at http://127.18.13.190:38115/ using document root <none> and password file <none>
I20240502 07:58:49.954741 25325 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:49.954806 25325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:49.954991 25325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:49.956796 25325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8803863bf7a5421a85a104bd413d2a57"
format_stamp: "Formatted at 2024-05-02 07:58:49 on dist-test-slave-f46z"
I20240502 07:58:49.957274 25325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8803863bf7a5421a85a104bd413d2a57"
format_stamp: "Formatted at 2024-05-02 07:58:49 on dist-test-slave-f46z"
I20240502 07:58:49.960716 25325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 07:58:49.963024 25339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:49.964123 25325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 07:58:49.964268 25325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8803863bf7a5421a85a104bd413d2a57"
format_stamp: "Formatted at 2024-05-02 07:58:49 on dist-test-slave-f46z"
I20240502 07:58:49.964368 25325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:49.977495 25325 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:49.978210 25325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:49.978380 25325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:49.985571 25325 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.190:46035
I20240502 07:58:49.985575 25391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.190:46035 every 8 connection(s)
I20240502 07:58:49.986660 25325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 07:58:49.989619 25392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:49.991955 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 25325
I20240502 07:58:49.992094 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 07:58:49.995209 25392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57: Bootstrap starting.
I20240502 07:58:49.997725 25392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:49.998617 25392 log.cc:824] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:50.000490 25392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57: No bootstrap required, opened a new log
I20240502 07:58:50.003130 25392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8803863bf7a5421a85a104bd413d2a57" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46035 } }
I20240502 07:58:50.003343 25392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:50.003401 25392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8803863bf7a5421a85a104bd413d2a57, State: Initialized, Role: FOLLOWER
I20240502 07:58:50.003818 25392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8803863bf7a5421a85a104bd413d2a57" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46035 } }
I20240502 07:58:50.003950 25392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:50.004000 25392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:50.004086 25392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:50.004853 25392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8803863bf7a5421a85a104bd413d2a57" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46035 } }
I20240502 07:58:50.005160 25392 leader_election.cc:302] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8803863bf7a5421a85a104bd413d2a57; no voters:
I20240502 07:58:50.005410 25392 leader_election.cc:288] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:58:50.005549 25397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:50.006175 25397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [term 1 LEADER]: Becoming Leader. State: Replica: 8803863bf7a5421a85a104bd413d2a57, State: Running, Role: LEADER
I20240502 07:58:50.006600 25392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:58:50.006578 25397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8803863bf7a5421a85a104bd413d2a57" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46035 } }
I20240502 07:58:50.008293 25399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8803863bf7a5421a85a104bd413d2a57. Latest consensus state: current_term: 1 leader_uuid: "8803863bf7a5421a85a104bd413d2a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8803863bf7a5421a85a104bd413d2a57" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46035 } } }
I20240502 07:58:50.008342 25398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8803863bf7a5421a85a104bd413d2a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8803863bf7a5421a85a104bd413d2a57" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 46035 } } }
I20240502 07:58:50.008400 25399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:50.008469 25398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57 [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:50.008890 25408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:58:50.011883 25408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:58:50.017263 25408 catalog_manager.cc:1347] Generated new cluster ID: 18b32d72fa664795a1f08070983b7542
I20240502 07:58:50.017371 25408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:58:50.031229 25408 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:58:50.032563 25408 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:58:50.046358 25408 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8803863bf7a5421a85a104bd413d2a57: Generated new TSK 0
I20240502 07:58:50.047317 25408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:58:50.053449 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.129:0
--local_ip_for_outbound_sockets=127.18.13.129
--webserver_interface=127.18.13.129
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--builtin_ntp_servers=127.18.13.148:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 07:58:50.171399 25416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:50.171651 25416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:50.171736 25416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:50.174897 25416 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:50.174993 25416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:50.175094 25416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.129
I20240502 07:58:50.178939 25416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.18.13.129
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--never_fsync=true
--heap_profile_path=/tmp/kudu.25416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.129
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:50.179960 25416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:50.181128 25416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:50.190101 25422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:50.191309 25421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:50.192260 25424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:50.193287 25416 server_base.cc:1034] running on GCE node
I20240502 07:58:50.193717 25416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:50.194350 25416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:50.195513 25416 hybrid_clock.cc:648] HybridClock initialized: now 1714636730195500 us; error 28 us; skew 500 ppm
I20240502 07:58:50.197696 25416 webserver.cc:460] Webserver started at http://127.18.13.129:41105/ using document root <none> and password file <none>
I20240502 07:58:50.198279 25416 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:50.198344 25416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:50.198524 25416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:50.200269 25416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:58:50.200733 25416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:58:50.204080 25416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 07:58:50.206321 25430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:50.207443 25416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:50.207553 25416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:58:50.207643 25416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:50.227924 25416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:50.228624 25416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:50.228804 25416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:50.229390 25416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:50.230376 25416 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:50.230427 25416 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:50.230492 25416 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:50.230518 25416 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:50.239166 25416 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.129:37377
I20240502 07:58:50.239209 25543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.129:37377 every 8 connection(s)
I20240502 07:58:50.240175 25416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 07:58:50.249138 25544 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46035
I20240502 07:58:50.249393 25544 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:50.250015 25544 heartbeater.cc:511] Master 127.18.13.190:46035 requested a full tablet report, sending...
I20240502 07:58:50.250124 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 25416
I20240502 07:58:50.250264 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 07:58:50.251631 25356 ts_manager.cc:194] Registered new tserver with Master: c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:58:50.252424 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.130:0
--local_ip_for_outbound_sockets=127.18.13.130
--webserver_interface=127.18.13.130
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--builtin_ntp_servers=127.18.13.148:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:50.252887 25356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.129:60359
W20240502 07:58:50.363042 25547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:50.363302 25547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:50.363349 25547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:50.366904 25547 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:50.366989 25547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:50.367060 25547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.130
I20240502 07:58:50.371085 25547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.18.13.130
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--never_fsync=true
--heap_profile_path=/tmp/kudu.25547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.130
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:50.372150 25547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:50.373291 25547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:50.383378 25555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:50.383780 25553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:50.384773 25552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:50.386246 25547 server_base.cc:1034] running on GCE node
I20240502 07:58:50.386718 25547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:50.387332 25547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:50.388490 25547 hybrid_clock.cc:648] HybridClock initialized: now 1714636730388475 us; error 33 us; skew 500 ppm
I20240502 07:58:50.390540 25547 webserver.cc:460] Webserver started at http://127.18.13.130:38409/ using document root <none> and password file <none>
I20240502 07:58:50.391139 25547 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:50.391206 25547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:50.391393 25547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:50.393141 25547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0dfa00c061c748b18772d5f0566944b5"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:58:50.393618 25547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0dfa00c061c748b18772d5f0566944b5"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:58:50.397138 25547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 07:58:50.399490 25561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:50.400624 25547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:58:50.400740 25547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0dfa00c061c748b18772d5f0566944b5"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:58:50.400849 25547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:50.415374 25547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:50.416064 25547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:50.416239 25547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:50.416803 25547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:50.417716 25547 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:50.417766 25547 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:50.417826 25547 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:50.417857 25547 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:50.426250 25547 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.130:38825
I20240502 07:58:50.426271 25674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.130:38825 every 8 connection(s)
I20240502 07:58:50.427253 25547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 07:58:50.429430 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 25547
I20240502 07:58:50.429563 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 07:58:50.431645 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.131:0
--local_ip_for_outbound_sockets=127.18.13.131
--webserver_interface=127.18.13.131
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--builtin_ntp_servers=127.18.13.148:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:50.436810 25675 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46035
I20240502 07:58:50.437067 25675 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:50.437654 25675 heartbeater.cc:511] Master 127.18.13.190:46035 requested a full tablet report, sending...
I20240502 07:58:50.438716 25356 ts_manager.cc:194] Registered new tserver with Master: 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:50.439271 25356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.130:38509
W20240502 07:58:50.546142 25678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:50.546391 25678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:50.546454 25678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:50.549733 25678 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:50.549820 25678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:50.549913 25678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.131
I20240502 07:58:50.553802 25678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.18.13.131
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--never_fsync=true
--heap_profile_path=/tmp/kudu.25678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.131
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:50.554849 25678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:50.556106 25678 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:50.565366 25684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:50.566491 25683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:50.567447 25686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:50.568535 25678 server_base.cc:1034] running on GCE node
I20240502 07:58:50.568980 25678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:50.569654 25678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:50.570806 25678 hybrid_clock.cc:648] HybridClock initialized: now 1714636730570787 us; error 43 us; skew 500 ppm
I20240502 07:58:50.573129 25678 webserver.cc:460] Webserver started at http://127.18.13.131:37601/ using document root <none> and password file <none>
I20240502 07:58:50.573732 25678 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:50.573794 25678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:50.573988 25678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:50.575726 25678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:58:50.576254 25678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:58:50.579878 25678 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 07:58:50.582240 25692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:50.583312 25678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:50.583420 25678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:58:50.583528 25678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:50.604952 25678 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:50.605650 25678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:50.605827 25678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:50.606446 25678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:50.607477 25678 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:50.607532 25678 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:50.607582 25678 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:50.607614 25678 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:50.616510 25678 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.131:42205
I20240502 07:58:50.616535 25805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.131:42205 every 8 connection(s)
I20240502 07:58:50.617535 25678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 07:58:50.619042 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 25678
I20240502 07:58:50.619184 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 07:58:50.621519 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.132:0
--local_ip_for_outbound_sockets=127.18.13.132
--webserver_interface=127.18.13.132
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--builtin_ntp_servers=127.18.13.148:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:50.627478 25806 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46035
I20240502 07:58:50.627729 25806 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:50.628373 25806 heartbeater.cc:511] Master 127.18.13.190:46035 requested a full tablet report, sending...
I20240502 07:58:50.629534 25356 ts_manager.cc:194] Registered new tserver with Master: 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:58:50.630347 25356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.131:43683
W20240502 07:58:50.735860 25809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:50.736135 25809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:50.736183 25809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:50.739374 25809 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:50.739470 25809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:50.739542 25809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.132
I20240502 07:58:50.743444 25809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.18.13.132
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--never_fsync=true
--heap_profile_path=/tmp/kudu.25809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.132
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:50.744515 25809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:50.745654 25809 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:50.755343 25814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:50.756019 25817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:50.757128 25815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:50.758214 25809 server_base.cc:1034] running on GCE node
I20240502 07:58:50.758718 25809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:50.759346 25809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:50.760527 25809 hybrid_clock.cc:648] HybridClock initialized: now 1714636730760503 us; error 39 us; skew 500 ppm
I20240502 07:58:50.762706 25809 webserver.cc:460] Webserver started at http://127.18.13.132:33837/ using document root <none> and password file <none>
I20240502 07:58:50.763288 25809 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:50.763353 25809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:50.763545 25809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:50.765324 25809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e9596bfb8d324b4da93a4288771ed177"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:58:50.765822 25809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e9596bfb8d324b4da93a4288771ed177"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:58:50.769502 25809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20240502 07:58:50.771835 25823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:50.772882 25809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240502 07:58:50.773001 25809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e9596bfb8d324b4da93a4288771ed177"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:58:50.773092 25809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:50.788867 25809 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:50.789480 25809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:50.789639 25809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:50.790180 25809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:50.791091 25809 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:50.791149 25809 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:50.791209 25809 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:50.791239 25809 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:50.799932 25809 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.132:45865
I20240502 07:58:50.799957 25936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.132:45865 every 8 connection(s)
I20240502 07:58:50.800935 25809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:58:50.807889 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 25809
I20240502 07:58:50.808036 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 07:58:50.809871 25937 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46035
I20240502 07:58:50.810127 25937 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:50.810345 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.133:0
--local_ip_for_outbound_sockets=127.18.13.133
--webserver_interface=127.18.13.133
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--builtin_ntp_servers=127.18.13.148:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:50.810808 25937 heartbeater.cc:511] Master 127.18.13.190:46035 requested a full tablet report, sending...
I20240502 07:58:50.811966 25356 ts_manager.cc:194] Registered new tserver with Master: e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:50.812538 25356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.132:39657
W20240502 07:58:50.922264 25940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:50.922541 25940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:50.922618 25940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:50.925761 25940 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:50.925848 25940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:50.925956 25940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.133
I20240502 07:58:50.929888 25940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.18.13.133
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--never_fsync=true
--heap_profile_path=/tmp/kudu.25940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.133
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:50.930974 25940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:50.932219 25940 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:50.941558 25945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:50.942155 25946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:50.943298 25948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:50.944350 25940 server_base.cc:1034] running on GCE node
I20240502 07:58:50.944824 25940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:50.945449 25940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:50.946607 25940 hybrid_clock.cc:648] HybridClock initialized: now 1714636730946587 us; error 41 us; skew 500 ppm
I20240502 07:58:50.948710 25940 webserver.cc:460] Webserver started at http://127.18.13.133:45141/ using document root <none> and password file <none>
I20240502 07:58:50.949306 25940 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:50.949369 25940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:50.949551 25940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:50.951381 25940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c5f3003b08fb4214b116c0be8dd62851"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:58:50.952133 25940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c5f3003b08fb4214b116c0be8dd62851"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:58:50.956089 25940 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 07:58:50.958632 25954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:50.959884 25940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:58:50.960019 25940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c5f3003b08fb4214b116c0be8dd62851"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:58:50.960125 25940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:50.976684 25940 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:50.977370 25940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:50.977520 25940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:50.978060 25940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:50.978998 25940 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:50.979049 25940 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:50.979092 25940 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:50.979166 25940 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:50.988524 25940 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.133:39097
I20240502 07:58:50.988540 26067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.133:39097 every 8 connection(s)
I20240502 07:58:50.989538 25940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:58:50.997175 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 25940
I20240502 07:58:50.997328 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 07:58:50.999939 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.134:0
--local_ip_for_outbound_sockets=127.18.13.134
--webserver_interface=127.18.13.134
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--builtin_ntp_servers=127.18.13.148:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:51.000702 26068 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46035
I20240502 07:58:51.000954 26068 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:51.001662 26068 heartbeater.cc:511] Master 127.18.13.190:46035 requested a full tablet report, sending...
I20240502 07:58:51.002805 25356 ts_manager.cc:194] Registered new tserver with Master: c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097)
I20240502 07:58:51.003415 25356 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.133:52617
W20240502 07:58:51.114079 26071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:51.114373 26071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:51.114423 26071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:51.117961 26071 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:51.118058 26071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:51.118157 26071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.134
I20240502 07:58:51.122231 26071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.18.13.134
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--never_fsync=true
--heap_profile_path=/tmp/kudu.26071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.134
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:51.123308 26071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:51.124572 26071 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:51.133590 26076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:51.134706 26079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:51.135648 26077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:51.136720 26071 server_base.cc:1034] running on GCE node
I20240502 07:58:51.137239 26071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:51.137920 26071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:51.139113 26071 hybrid_clock.cc:648] HybridClock initialized: now 1714636731139093 us; error 33 us; skew 500 ppm
I20240502 07:58:51.141361 26071 webserver.cc:460] Webserver started at http://127.18.13.134:42073/ using document root <none> and password file <none>
I20240502 07:58:51.141928 26071 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:51.141988 26071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:51.142177 26071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:51.143999 26071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "a17bbff6a71b4667b14f5dd12bca3057"
format_stamp: "Formatted at 2024-05-02 07:58:51 on dist-test-slave-f46z"
I20240502 07:58:51.145368 26071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "a17bbff6a71b4667b14f5dd12bca3057"
format_stamp: "Formatted at 2024-05-02 07:58:51 on dist-test-slave-f46z"
I20240502 07:58:51.149353 26071 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 07:58:51.151860 26085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:51.152962 26071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:58:51.153092 26071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a17bbff6a71b4667b14f5dd12bca3057"
format_stamp: "Formatted at 2024-05-02 07:58:51 on dist-test-slave-f46z"
I20240502 07:58:51.153223 26071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:51.175303 26071 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:51.176151 26071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:51.176373 26071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:51.176982 26071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:51.177999 26071 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:51.178062 26071 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:51.178121 26071 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:51.178158 26071 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:51.187964 26071 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.134:43037
I20240502 07:58:51.187968 26198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.134:43037 every 8 connection(s)
I20240502 07:58:51.189136 26071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 07:58:51.196877 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 26071
I20240502 07:58:51.197007 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 07:58:51.198810 26199 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46035
I20240502 07:58:51.199102 26199 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:51.199203 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.135:0
--local_ip_for_outbound_sockets=127.18.13.135
--webserver_interface=127.18.13.135
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--builtin_ntp_servers=127.18.13.148:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:51.199743 26199 heartbeater.cc:511] Master 127.18.13.190:46035 requested a full tablet report, sending...
I20240502 07:58:51.200868 25354 ts_manager.cc:194] Registered new tserver with Master: a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037)
I20240502 07:58:51.201468 25354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.134:60503
I20240502 07:58:51.255241 25544 heartbeater.cc:503] Master 127.18.13.190:46035 was elected leader, sending a full tablet report...
W20240502 07:58:51.313720 26202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:51.313984 26202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:51.314031 26202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:51.317273 26202 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:51.317359 26202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:51.317448 26202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.135
I20240502 07:58:51.321377 26202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.18.13.135
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--never_fsync=true
--heap_profile_path=/tmp/kudu.26202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.135
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:51.322425 26202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:51.323619 26202 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:51.332791 26207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:51.333173 26208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:51.334265 26210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:51.335657 26202 server_base.cc:1034] running on GCE node
I20240502 07:58:51.336208 26202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:51.336908 26202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:51.338105 26202 hybrid_clock.cc:648] HybridClock initialized: now 1714636731338093 us; error 31 us; skew 500 ppm
I20240502 07:58:51.345849 26202 webserver.cc:460] Webserver started at http://127.18.13.135:33083/ using document root <none> and password file <none>
I20240502 07:58:51.346503 26202 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:51.346578 26202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:51.346786 26202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:51.348690 26202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "c6746058199446968d2496df55cdee42"
format_stamp: "Formatted at 2024-05-02 07:58:51 on dist-test-slave-f46z"
I20240502 07:58:51.349269 26202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "c6746058199446968d2496df55cdee42"
format_stamp: "Formatted at 2024-05-02 07:58:51 on dist-test-slave-f46z"
I20240502 07:58:51.353005 26202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 07:58:51.355621 26216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:51.356861 26202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:58:51.356997 26202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c6746058199446968d2496df55cdee42"
format_stamp: "Formatted at 2024-05-02 07:58:51 on dist-test-slave-f46z"
I20240502 07:58:51.357125 26202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:51.371990 26202 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:51.372691 26202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:51.372915 26202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:51.373495 26202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:51.374631 26202 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:51.374691 26202 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:51.374758 26202 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:51.374783 26202 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:51.383922 26202 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.135:34615
I20240502 07:58:51.383936 26329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.135:34615 every 8 connection(s)
I20240502 07:58:51.384964 26202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 07:58:51.385769 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 26202
I20240502 07:58:51.385900 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 07:58:51.388490 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.136:0
--local_ip_for_outbound_sockets=127.18.13.136
--webserver_interface=127.18.13.136
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--builtin_ntp_servers=127.18.13.148:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:58:51.394748 26330 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46035
I20240502 07:58:51.395093 26330 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:51.396000 26330 heartbeater.cc:511] Master 127.18.13.190:46035 requested a full tablet report, sending...
I20240502 07:58:51.397408 25354 ts_manager.cc:194] Registered new tserver with Master: c6746058199446968d2496df55cdee42 (127.18.13.135:34615)
I20240502 07:58:51.397994 25354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.135:53351
I20240502 07:58:51.441291 25675 heartbeater.cc:503] Master 127.18.13.190:46035 was elected leader, sending a full tablet report...
W20240502 07:58:51.511363 26333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:51.511698 26333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:51.511816 26333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:51.516273 26333 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:51.516378 26333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:51.516495 26333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.136
I20240502 07:58:51.521605 26333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.18.13.136
--webserver_port=0
--tserver_master_addrs=127.18.13.190:46035
--never_fsync=true
--heap_profile_path=/tmp/kudu.26333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.136
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:51.522807 26333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:51.524243 26333 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:51.534037 26338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:51.535225 26341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:51.536391 26339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:51.537464 26333 server_base.cc:1034] running on GCE node
I20240502 07:58:51.537931 26333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:51.538564 26333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:51.539743 26333 hybrid_clock.cc:648] HybridClock initialized: now 1714636731539717 us; error 53 us; skew 500 ppm
I20240502 07:58:51.541913 26333 webserver.cc:460] Webserver started at http://127.18.13.136:40371/ using document root <none> and password file <none>
I20240502 07:58:51.542591 26333 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:51.542670 26333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:51.542901 26333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:51.545234 26333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "85fbc6c98f0643469c92215a5cfb2e38"
format_stamp: "Formatted at 2024-05-02 07:58:51 on dist-test-slave-f46z"
I20240502 07:58:51.545890 26333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "85fbc6c98f0643469c92215a5cfb2e38"
format_stamp: "Formatted at 2024-05-02 07:58:51 on dist-test-slave-f46z"
I20240502 07:58:51.549930 26333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 07:58:51.552390 26347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:51.553431 26333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:58:51.553575 26333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "85fbc6c98f0643469c92215a5cfb2e38"
format_stamp: "Formatted at 2024-05-02 07:58:51 on dist-test-slave-f46z"
I20240502 07:58:51.553669 26333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:51.582988 26333 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:51.583657 26333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:51.583874 26333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:51.584470 26333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:51.585450 26333 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:51.585501 26333 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:51.585561 26333 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:51.585603 26333 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:51.594956 26333 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.136:38043
I20240502 07:58:51.594974 26460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.136:38043 every 8 connection(s)
I20240502 07:58:51.596084 26333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 07:58:51.604997 26461 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46035
I20240502 07:58:51.605257 26461 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:51.605329 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 26333
I20240502 07:58:51.605468 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 07:58:51.605867 26461 heartbeater.cc:511] Master 127.18.13.190:46035 requested a full tablet report, sending...
I20240502 07:58:51.606956 25354 ts_manager.cc:194] Registered new tserver with Master: 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043)
I20240502 07:58:51.607836 25354 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.136:47649
I20240502 07:58:51.608075 18486 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 07:58:51.618711 18486 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 07:58:51.618813 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 25940
I20240502 07:58:51.625310 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 26071
I20240502 07:58:51.630990 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 26202
I20240502 07:58:51.632593 25806 heartbeater.cc:503] Master 127.18.13.190:46035 was elected leader, sending a full tablet report...
I20240502 07:58:51.636724 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 26333
I20240502 07:58:51.814488 25937 heartbeater.cc:503] Master 127.18.13.190:46035 was elected leader, sending a full tablet report...
I20240502 07:58:55.393898 25354 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40498:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:55.415578 25870 tablet_service.cc:1467] Processing CreateTablet for tablet ff31ddd6b3d74a9aae6617afb56c40f5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=616a1d4f85774bf6b08fc4da5a27702a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:55.415577 25476 tablet_service.cc:1467] Processing CreateTablet for tablet ff31ddd6b3d74a9aae6617afb56c40f5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=616a1d4f85774bf6b08fc4da5a27702a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:55.416018 25608 tablet_service.cc:1467] Processing CreateTablet for tablet ee25634bf534443b90fdbdcebdbcb02e (DEFAULT_TABLE table=rebalance_test_table_0 [id=616a1d4f85774bf6b08fc4da5a27702a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:55.415578 25478 tablet_service.cc:1467] Processing CreateTablet for tablet 5a0605c8c64f4c6fb5e7315a118a0d7f (DEFAULT_TABLE table=rebalance_test_table_0 [id=616a1d4f85774bf6b08fc4da5a27702a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:55.416658 25476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff31ddd6b3d74a9aae6617afb56c40f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:55.416661 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff31ddd6b3d74a9aae6617afb56c40f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:55.416750 25608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee25634bf534443b90fdbdcebdbcb02e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:55.417080 25478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a0605c8c64f4c6fb5e7315a118a0d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:55.419462 25871 tablet_service.cc:1467] Processing CreateTablet for tablet ee25634bf534443b90fdbdcebdbcb02e (DEFAULT_TABLE table=rebalance_test_table_0 [id=616a1d4f85774bf6b08fc4da5a27702a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:55.419770 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee25634bf534443b90fdbdcebdbcb02e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:55.420126 25477 tablet_service.cc:1467] Processing CreateTablet for tablet ee25634bf534443b90fdbdcebdbcb02e (DEFAULT_TABLE table=rebalance_test_table_0 [id=616a1d4f85774bf6b08fc4da5a27702a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:55.420390 25477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee25634bf534443b90fdbdcebdbcb02e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:55.421870 25609 tablet_service.cc:1467] Processing CreateTablet for tablet 5a0605c8c64f4c6fb5e7315a118a0d7f (DEFAULT_TABLE table=rebalance_test_table_0 [id=616a1d4f85774bf6b08fc4da5a27702a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:55.422133 25609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a0605c8c64f4c6fb5e7315a118a0d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:55.426632 26483 tablet_bootstrap.cc:492] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:55.426981 26482 tablet_bootstrap.cc:492] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:55.428717 25740 tablet_service.cc:1467] Processing CreateTablet for tablet 5a0605c8c64f4c6fb5e7315a118a0d7f (DEFAULT_TABLE table=rebalance_test_table_0 [id=616a1d4f85774bf6b08fc4da5a27702a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:55.429263 26482 tablet_bootstrap.cc:654] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:55.429654 26483 tablet_bootstrap.cc:654] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:55.429729 25740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a0605c8c64f4c6fb5e7315a118a0d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:55.430078 26482 log.cc:824] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:55.430737 26483 log.cc:824] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:55.430878 26481 tablet_bootstrap.cc:492] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:55.432096 25739 tablet_service.cc:1467] Processing CreateTablet for tablet ff31ddd6b3d74a9aae6617afb56c40f5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=616a1d4f85774bf6b08fc4da5a27702a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:55.432354 25739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff31ddd6b3d74a9aae6617afb56c40f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:55.432941 26481 tablet_bootstrap.cc:654] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:55.433727 26481 log.cc:824] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:55.433975 26483 tablet_bootstrap.cc:492] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:55.434170 26483 ts_tablet_manager.cc:1367] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 07:58:55.435058 26482 tablet_bootstrap.cc:492] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:55.435281 26482 ts_tablet_manager.cc:1367] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240502 07:58:55.437357 26481 tablet_bootstrap.cc:492] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:55.437554 26481 ts_tablet_manager.cc:1367] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 07:58:55.437518 26483 raft_consensus.cc:348] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.437752 26483 raft_consensus.cc:374] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:55.437816 26483 raft_consensus.cc:729] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:55.438738 26487 tablet_bootstrap.cc:492] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:55.438998 26482 raft_consensus.cc:348] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.439231 26482 raft_consensus.cc:374] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:55.439298 26482 raft_consensus.cc:729] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:55.439337 26483 consensus_queue.cc:260] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.439778 26482 consensus_queue.cc:260] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.440339 26483 ts_tablet_manager.cc:1398] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 07:58:55.440645 26482 ts_tablet_manager.cc:1398] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 07:58:55.440732 26483 tablet_bootstrap.cc:492] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:55.441040 26482 tablet_bootstrap.cc:492] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:55.441795 26481 raft_consensus.cc:348] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.442057 26483 tablet_bootstrap.cc:654] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:55.442015 26481 raft_consensus.cc:374] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:55.442142 26481 raft_consensus.cc:729] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:55.442317 26482 tablet_bootstrap.cc:654] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:55.442674 26481 consensus_queue.cc:260] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.443615 26481 ts_tablet_manager.cc:1398] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 07:58:55.443732 26487 tablet_bootstrap.cc:654] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:55.444706 26487 log.cc:824] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:55.445399 26483 tablet_bootstrap.cc:492] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:55.445518 26483 ts_tablet_manager.cc:1367] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:58:55.446147 26483 raft_consensus.cc:348] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.446264 26483 raft_consensus.cc:374] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:55.446292 26483 raft_consensus.cc:729] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:55.446403 26483 consensus_queue.cc:260] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.446703 26483 ts_tablet_manager.cc:1398] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:55.446923 26481 tablet_bootstrap.cc:492] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:55.447103 26487 tablet_bootstrap.cc:492] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:55.447314 26487 ts_tablet_manager.cc:1367] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20240502 07:58:55.448029 26482 tablet_bootstrap.cc:492] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:55.448055 26481 tablet_bootstrap.cc:654] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:55.448123 26482 ts_tablet_manager.cc:1367] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 07:58:55.448697 26482 raft_consensus.cc:348] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.448817 26482 raft_consensus.cc:374] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:55.448858 26482 raft_consensus.cc:729] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:55.449005 26482 consensus_queue.cc:260] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.449347 26482 ts_tablet_manager.cc:1398] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:55.449512 26482 tablet_bootstrap.cc:492] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:55.450439 26482 tablet_bootstrap.cc:654] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:55.450500 26481 tablet_bootstrap.cc:492] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:55.450584 26481 ts_tablet_manager.cc:1367] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:55.451148 26481 raft_consensus.cc:348] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:55.451295 26481 raft_consensus.cc:374] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:55.451339 26481 raft_consensus.cc:729] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:55.451463 26481 consensus_queue.cc:260] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:55.451493 26487 raft_consensus.cc:348] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.451656 26482 tablet_bootstrap.cc:492] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:55.451726 26487 raft_consensus.cc:374] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:55.451742 26482 ts_tablet_manager.cc:1367] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:55.451748 26481 ts_tablet_manager.cc:1398] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:55.451807 26487 raft_consensus.cc:729] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:55.452333 26482 raft_consensus.cc:348] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:55.452420 26482 raft_consensus.cc:374] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:55.452304 26487 consensus_queue.cc:260] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.452440 26482 raft_consensus.cc:729] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:55.452546 26482 consensus_queue.cc:260] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:55.453318 26487 ts_tablet_manager.cc:1398] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 07:58:55.453393 26482 ts_tablet_manager.cc:1398] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:55.453583 26487 tablet_bootstrap.cc:492] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:55.454633 26487 tablet_bootstrap.cc:654] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:55.456209 26487 tablet_bootstrap.cc:492] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:55.456296 26487 ts_tablet_manager.cc:1367] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 07:58:55.456817 26487 raft_consensus.cc:348] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:55.456914 26487 raft_consensus.cc:374] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:55.456947 26487 raft_consensus.cc:729] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:55.457095 26487 consensus_queue.cc:260] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:55.457345 26487 ts_tablet_manager.cc:1398] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 07:58:55.495409 25545 tablet.cc:2369] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 07:58:55.555646 25938 tablet.cc:2369] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:55.557348 26489 raft_consensus.cc:482] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:55.557560 26489 raft_consensus.cc:504] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.558657 26489 leader_election.cc:288] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:55.562840 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee25634bf534443b90fdbdcebdbcb02e" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:55.563213 25629 raft_consensus.cc:2454] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:55.563275 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee25634bf534443b90fdbdcebdbcb02e" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:58:55.563556 25891 raft_consensus.cc:2454] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:55.563655 25434 leader_election.cc:302] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:55.563967 26489 raft_consensus.cc:2781] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:55.564057 26489 raft_consensus.cc:482] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:55.564098 26489 raft_consensus.cc:3037] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:55.564999 26489 raft_consensus.cc:504] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.565539 26489 leader_election.cc:288] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Requested vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:55.565897 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee25634bf534443b90fdbdcebdbcb02e" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:58:55.566033 25891 raft_consensus.cc:3037] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:55.566097 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee25634bf534443b90fdbdcebdbcb02e" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:55.566238 25629 raft_consensus.cc:3037] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:55.566970 25891 raft_consensus.cc:2454] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:55.567116 25629 raft_consensus.cc:2454] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:55.567319 25433 leader_election.cc:302] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3be11f0fe304e0f8e008dc1e4c41f70, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:55.567498 26489 raft_consensus.cc:2781] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:55.567731 26489 raft_consensus.cc:686] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Becoming Leader. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Running, Role: LEADER
I20240502 07:58:55.568111 26489 consensus_queue.cc:237] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.571354 25354 catalog_manager.cc:5484] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:55.622202 25807 tablet.cc:2369] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:55.669536 26489 raft_consensus.cc:482] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:55.669716 26489 raft_consensus.cc:504] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:55.670258 26489 leader_election.cc:288] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:58:55.670711 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff31ddd6b3d74a9aae6617afb56c40f5" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:58:55.670888 25891 raft_consensus.cc:2454] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:55.671241 25433 leader_election.cc:302] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3be11f0fe304e0f8e008dc1e4c41f70, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:55.671432 26489 raft_consensus.cc:2781] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:55.671489 26489 raft_consensus.cc:482] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:55.671615 26489 raft_consensus.cc:3037] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:55.672750 26489 raft_consensus.cc:504] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:55.673164 26489 leader_election.cc:288] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Requested vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:58:55.673732 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff31ddd6b3d74a9aae6617afb56c40f5" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:58:55.673866 25891 raft_consensus.cc:3037] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:55.674564 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff31ddd6b3d74a9aae6617afb56c40f5" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
I20240502 07:58:55.674567 25760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff31ddd6b3d74a9aae6617afb56c40f5" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" is_pre_election: true
I20240502 07:58:55.674789 25891 raft_consensus.cc:2454] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:55.674835 25759 raft_consensus.cc:3037] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:55.675102 25433 leader_election.cc:302] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3be11f0fe304e0f8e008dc1e4c41f70, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:55.675488 26489 raft_consensus.cc:2781] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:55.675576 26489 raft_consensus.cc:686] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Becoming Leader. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Running, Role: LEADER
I20240502 07:58:55.675734 26489 consensus_queue.cc:237] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:55.676007 25759 raft_consensus.cc:2454] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:55.677592 25354 catalog_manager.cc:5484] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:55.682449 25676 tablet.cc:2369] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:55.811825 26489 raft_consensus.cc:482] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:55.811973 26489 raft_consensus.cc:504] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.812433 26489 leader_election.cc:288] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:55.812880 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a0605c8c64f4c6fb5e7315a118a0d7f" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:55.812889 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a0605c8c64f4c6fb5e7315a118a0d7f" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" is_pre_election: true
I20240502 07:58:55.813017 25629 raft_consensus.cc:2454] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:55.813038 25759 raft_consensus.cc:2454] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:55.813342 25432 leader_election.cc:302] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:55.813544 26489 raft_consensus.cc:2781] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:55.813606 26489 raft_consensus.cc:482] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:55.813635 26489 raft_consensus.cc:3037] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:55.814455 26489 raft_consensus.cc:504] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.814796 26489 leader_election.cc:288] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Requested vote from peers 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:55.815110 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a0605c8c64f4c6fb5e7315a118a0d7f" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
I20240502 07:58:55.815125 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a0605c8c64f4c6fb5e7315a118a0d7f" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:55.815224 25629 raft_consensus.cc:3037] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:55.815233 25759 raft_consensus.cc:3037] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:55.816010 25759 raft_consensus.cc:2454] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:55.816012 25629 raft_consensus.cc:2454] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:55.816287 25434 leader_election.cc:302] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:55.816431 26489 raft_consensus.cc:2781] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:55.816516 26489 raft_consensus.cc:686] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Becoming Leader. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Running, Role: LEADER
I20240502 07:58:55.816654 26489 consensus_queue.cc:237] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:55.818295 25354 catalog_manager.cc:5484] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:55.951332 26489 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:55.957851 26508 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 07:58:55.978683 26504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:55.978957 26504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:55.982201 26504 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 07:58:56.074450 25629 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:56.074731 25759 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:56.074966 26500 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:56.075227 25760 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:56.075740 25891 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:56.076529 26508 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:56.077313 26489 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:56.077360 26508 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:56.091480 26509 mvcc.cc:205] Tried to move back new op lower bound from 7023152070954758144 to 7023152069906264064. Current Snapshot: MvccSnapshot[applied={T|T < 7023152070954758144}]
I20240502 07:58:56.102236 26512 mvcc.cc:205] Tried to move back new op lower bound from 7023152070957764608 to 7023152069329575936. Current Snapshot: MvccSnapshot[applied={T|T < 7023152070957764608}]
I20240502 07:58:56.103873 26539 mvcc.cc:205] Tried to move back new op lower bound from 7023152070954758144 to 7023152069906264064. Current Snapshot: MvccSnapshot[applied={T|T < 7023152070954758144 or (T in {7023152070954758144})}]
I20240502 07:58:56.119449 26549 mvcc.cc:205] Tried to move back new op lower bound from 7023152070954758144 to 7023152069906264064. Current Snapshot: MvccSnapshot[applied={T|T < 7023152070954758144}]
INSERT report
rows total: 2000
time total: 112.292 ms
time per row: 0.056146 ms
I20240502 07:58:56.144457 25349 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40498:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:56.150377 25739 tablet_service.cc:1467] Processing CreateTablet for tablet 87d6266008784883891ca32c01363dbe (DEFAULT_TABLE table=rebalance_test_table_1 [id=a5f087acc4d94b0f9694fb1cf95e7dca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.150456 25870 tablet_service.cc:1467] Processing CreateTablet for tablet 87d6266008784883891ca32c01363dbe (DEFAULT_TABLE table=rebalance_test_table_1 [id=a5f087acc4d94b0f9694fb1cf95e7dca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.150722 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87d6266008784883891ca32c01363dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.150707 25740 tablet_service.cc:1467] Processing CreateTablet for tablet 71a67cb0f6b54bfb9eb44400fe21ce40 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a5f087acc4d94b0f9694fb1cf95e7dca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.151006 25609 tablet_service.cc:1467] Processing CreateTablet for tablet 87d6266008784883891ca32c01363dbe (DEFAULT_TABLE table=rebalance_test_table_1 [id=a5f087acc4d94b0f9694fb1cf95e7dca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.151298 25609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87d6266008784883891ca32c01363dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.151443 25738 tablet_service.cc:1467] Processing CreateTablet for tablet 9722dae4e4414084bdd8066b41bcc47a (DEFAULT_TABLE table=rebalance_test_table_1 [id=a5f087acc4d94b0f9694fb1cf95e7dca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.150722 25739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87d6266008784883891ca32c01363dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.153544 26568 tablet_bootstrap.cc:492] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:56.154022 25608 tablet_service.cc:1467] Processing CreateTablet for tablet 9722dae4e4414084bdd8066b41bcc47a (DEFAULT_TABLE table=rebalance_test_table_1 [id=a5f087acc4d94b0f9694fb1cf95e7dca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.154273 25608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9722dae4e4414084bdd8066b41bcc47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.154314 25476 tablet_service.cc:1467] Processing CreateTablet for tablet 9722dae4e4414084bdd8066b41bcc47a (DEFAULT_TABLE table=rebalance_test_table_1 [id=a5f087acc4d94b0f9694fb1cf95e7dca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.154603 25476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9722dae4e4414084bdd8066b41bcc47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.154754 25738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9722dae4e4414084bdd8066b41bcc47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.154858 26568 tablet_bootstrap.cc:654] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.156096 25870 tablet_service.cc:1467] Processing CreateTablet for tablet 71a67cb0f6b54bfb9eb44400fe21ce40 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a5f087acc4d94b0f9694fb1cf95e7dca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.156378 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a67cb0f6b54bfb9eb44400fe21ce40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.156857 26571 tablet_bootstrap.cc:492] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:56.157028 25740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a67cb0f6b54bfb9eb44400fe21ce40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.157320 25609 tablet_service.cc:1467] Processing CreateTablet for tablet 71a67cb0f6b54bfb9eb44400fe21ce40 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a5f087acc4d94b0f9694fb1cf95e7dca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.157543 25609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a67cb0f6b54bfb9eb44400fe21ce40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.157953 26571 tablet_bootstrap.cc:654] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.159009 26571 tablet_bootstrap.cc:492] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:56.159144 26571 ts_tablet_manager.cc:1367] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:56.159559 26571 raft_consensus.cc:348] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.159634 26569 tablet_bootstrap.cc:492] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:56.159684 26571 raft_consensus.cc:374] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.159719 26571 raft_consensus.cc:729] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.159875 26571 consensus_queue.cc:260] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.160125 26571 ts_tablet_manager.cc:1398] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:56.160667 26569 tablet_bootstrap.cc:654] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.160704 26567 tablet_bootstrap.cc:492] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:56.161936 26568 tablet_bootstrap.cc:492] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:56.161952 26569 tablet_bootstrap.cc:492] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:56.162027 26568 ts_tablet_manager.cc:1367] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 07:58:56.162034 26569 ts_tablet_manager.cc:1367] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:56.162567 26567 tablet_bootstrap.cc:654] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.162528 26568 raft_consensus.cc:348] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:56.162549 26569 raft_consensus.cc:348] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:56.162639 26568 raft_consensus.cc:374] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.162653 26569 raft_consensus.cc:374] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.162685 26568 raft_consensus.cc:729] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.162690 26569 raft_consensus.cc:729] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.162813 26568 consensus_queue.cc:260] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:56.162873 26569 consensus_queue.cc:260] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:56.163120 26568 ts_tablet_manager.cc:1398] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:56.163296 26568 tablet_bootstrap.cc:492] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:56.163570 26569 ts_tablet_manager.cc:1398] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:56.163733 26569 tablet_bootstrap.cc:492] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:56.164203 26568 tablet_bootstrap.cc:654] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.164544 26567 tablet_bootstrap.cc:492] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:56.164625 26567 ts_tablet_manager.cc:1367] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:56.164732 26569 tablet_bootstrap.cc:654] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.165117 26567 raft_consensus.cc:348] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:56.165256 26567 raft_consensus.cc:374] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.165297 26567 raft_consensus.cc:729] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.165444 26567 consensus_queue.cc:260] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:56.165712 26567 ts_tablet_manager.cc:1398] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:56.165877 26567 tablet_bootstrap.cc:492] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:56.166817 26567 tablet_bootstrap.cc:654] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.167156 26569 tablet_bootstrap.cc:492] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:56.167228 26569 ts_tablet_manager.cc:1367] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:56.167529 26568 tablet_bootstrap.cc:492] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:56.167610 26568 ts_tablet_manager.cc:1367] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 07:58:56.167757 26569 raft_consensus.cc:348] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.167868 26567 tablet_bootstrap.cc:492] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:56.167891 26569 raft_consensus.cc:374] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.167922 26569 raft_consensus.cc:729] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.167940 26567 ts_tablet_manager.cc:1367] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:56.168062 26569 consensus_queue.cc:260] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.168120 26568 raft_consensus.cc:348] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.168215 26568 raft_consensus.cc:374] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.168249 26568 raft_consensus.cc:729] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.168334 26569 ts_tablet_manager.cc:1398] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:56.168377 26567 raft_consensus.cc:348] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.168463 26567 raft_consensus.cc:374] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.168501 26569 tablet_bootstrap.cc:492] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:56.168493 26567 raft_consensus.cc:729] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.168604 26567 consensus_queue.cc:260] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.168869 26567 ts_tablet_manager.cc:1398] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:56.169009 26568 consensus_queue.cc:260] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.169358 26568 ts_tablet_manager.cc:1398] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:58:56.169551 26568 tablet_bootstrap.cc:492] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:56.169705 26569 tablet_bootstrap.cc:654] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.170473 26568 tablet_bootstrap.cc:654] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.171824 26568 tablet_bootstrap.cc:492] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:56.171923 26568 ts_tablet_manager.cc:1367] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 07:58:56.172384 26568 raft_consensus.cc:348] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.172497 26568 raft_consensus.cc:374] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.172533 26568 raft_consensus.cc:729] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.172650 26569 tablet_bootstrap.cc:492] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:56.172643 26568 consensus_queue.cc:260] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.172749 26569 ts_tablet_manager.cc:1367] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:56.172930 26568 ts_tablet_manager.cc:1398] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:56.173141 26569 raft_consensus.cc:348] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.173231 26569 raft_consensus.cc:374] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.173271 26569 raft_consensus.cc:729] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.173391 26569 consensus_queue.cc:260] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.173651 26569 ts_tablet_manager.cc:1398] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:56.203613 26562 raft_consensus.cc:482] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:56.203780 26562 raft_consensus.cc:504] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.205086 26562 leader_election.cc:288] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:56.209909 25628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a67cb0f6b54bfb9eb44400fe21ce40" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:56.209980 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a67cb0f6b54bfb9eb44400fe21ce40" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:58:56.210106 25628 raft_consensus.cc:2454] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:56.210131 25890 raft_consensus.cc:2454] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:56.210527 25695 leader_election.cc:302] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:56.210824 26562 raft_consensus.cc:2781] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:56.210889 26562 raft_consensus.cc:482] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:56.210934 26562 raft_consensus.cc:3037] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:56.211892 26562 raft_consensus.cc:504] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.212272 26562 leader_election.cc:288] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Requested vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:56.212615 25628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a67cb0f6b54bfb9eb44400fe21ce40" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:56.212618 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a67cb0f6b54bfb9eb44400fe21ce40" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:58:56.212723 25890 raft_consensus.cc:3037] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:56.212733 25628 raft_consensus.cc:3037] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:56.213521 25628 raft_consensus.cc:2454] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:58:56.213523 25890 raft_consensus.cc:2454] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:58:56.213793 25695 leader_election.cc:302] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:56.213941 26562 raft_consensus.cc:2781] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:56.214032 26562 raft_consensus.cc:686] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Becoming Leader. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Running, Role: LEADER
I20240502 07:58:56.214332 26562 consensus_queue.cc:237] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.216802 25351 catalog_manager.cc:5484] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: term changed from 0 to 1, leader changed from <none> to 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131). New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:56.225016 26562 raft_consensus.cc:482] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:56.225132 26562 raft_consensus.cc:504] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.225569 26562 leader_election.cc:288] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:56.226230 25628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9722dae4e4414084bdd8066b41bcc47a" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:56.226369 25628 raft_consensus.cc:2454] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:56.226655 25696 leader_election.cc:302] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, 14d8fd1fc3a04d9d967b8033df84f34a; no voters:
I20240502 07:58:56.226824 26562 raft_consensus.cc:2781] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:56.226868 26562 raft_consensus.cc:482] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:56.226913 26562 raft_consensus.cc:3037] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:56.227689 26562 raft_consensus.cc:504] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.228032 26562 leader_election.cc:288] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Requested vote from peers c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:56.228403 25628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9722dae4e4414084bdd8066b41bcc47a" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:56.228535 25628 raft_consensus.cc:3037] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:56.229065 25498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9722dae4e4414084bdd8066b41bcc47a" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" is_pre_election: true
I20240502 07:58:56.229065 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9722dae4e4414084bdd8066b41bcc47a" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
I20240502 07:58:56.229313 25628 raft_consensus.cc:2454] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:58:56.229328 25498 raft_consensus.cc:2454] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:56.229678 25696 leader_election.cc:302] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, 14d8fd1fc3a04d9d967b8033df84f34a; no voters:
I20240502 07:58:56.229864 26562 raft_consensus.cc:2781] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:56.229954 26562 raft_consensus.cc:686] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Becoming Leader. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Running, Role: LEADER
I20240502 07:58:56.230095 26562 consensus_queue.cc:237] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.231695 25351 catalog_manager.cc:5484] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: term changed from 0 to 1, leader changed from <none> to 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131). New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:56.382346 26562 raft_consensus.cc:482] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:56.382495 26562 raft_consensus.cc:504] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:56.382963 26562 leader_election.cc:288] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:56.383469 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d6266008784883891ca32c01363dbe" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:58:56.383481 25628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d6266008784883891ca32c01363dbe" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:56.383603 25890 raft_consensus.cc:2454] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:56.383633 25628 raft_consensus.cc:2454] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:56.383940 25695 leader_election.cc:302] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:56.384114 26562 raft_consensus.cc:2781] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:56.384188 26562 raft_consensus.cc:482] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:56.384224 26562 raft_consensus.cc:3037] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:56.385057 26562 raft_consensus.cc:504] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:56.385414 26562 leader_election.cc:288] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Requested vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:56.385857 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d6266008784883891ca32c01363dbe" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:58:56.385857 25628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d6266008784883891ca32c01363dbe" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:56.385998 25890 raft_consensus.cc:3037] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:56.385998 25628 raft_consensus.cc:3037] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:56.386979 25890 raft_consensus.cc:2454] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:58:56.386979 25628 raft_consensus.cc:2454] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:58:56.387310 25695 leader_election.cc:302] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:56.387519 26562 raft_consensus.cc:2781] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:56.387609 26562 raft_consensus.cc:686] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Becoming Leader. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Running, Role: LEADER
I20240502 07:58:56.387763 26562 consensus_queue.cc:237] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:56.389555 25351 catalog_manager.cc:5484] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: term changed from 0 to 1, leader changed from <none> to 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131). New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:56.509280 26587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:56.509579 26587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:56.512921 26587 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 07:58:56.600286 25628 raft_consensus.cc:1261] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:56.600694 25497 raft_consensus.cc:3037] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:56.600317 25629 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:56.600872 25890 raft_consensus.cc:1261] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:56.601490 26548 consensus_queue.cc:1035] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:56.601706 26562 consensus_queue.cc:1035] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:56.602023 25497 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:56.602068 26562 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:56.603722 26552 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:56.609676 25891 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:56.609961 25627 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:56.610446 26552 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:56.613437 26548 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 107.5 ms
time per row: 0.0537498 ms
I20240502 07:58:56.678830 25353 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40498:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:56.684420 25870 tablet_service.cc:1467] Processing CreateTablet for tablet 056ba8d15d18409e946c905b8876097a (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9eeb74ff004488ba3049c0e8e535bb5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.684420 25476 tablet_service.cc:1467] Processing CreateTablet for tablet 056ba8d15d18409e946c905b8876097a (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9eeb74ff004488ba3049c0e8e535bb5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.684774 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056ba8d15d18409e946c905b8876097a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.684773 25476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056ba8d15d18409e946c905b8876097a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.685602 25740 tablet_service.cc:1467] Processing CreateTablet for tablet a1fa713b432740d98bcab918d251b54d (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9eeb74ff004488ba3049c0e8e535bb5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.685797 25609 tablet_service.cc:1467] Processing CreateTablet for tablet 056ba8d15d18409e946c905b8876097a (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9eeb74ff004488ba3049c0e8e535bb5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.685912 25740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fa713b432740d98bcab918d251b54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.686043 25609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056ba8d15d18409e946c905b8876097a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.687294 25608 tablet_service.cc:1467] Processing CreateTablet for tablet c95cd9642c1c46cabb23f6c62c6ad4f5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9eeb74ff004488ba3049c0e8e535bb5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.687536 25608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95cd9642c1c46cabb23f6c62c6ad4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.688107 25870 tablet_service.cc:1467] Processing CreateTablet for tablet a1fa713b432740d98bcab918d251b54d (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9eeb74ff004488ba3049c0e8e535bb5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.688174 25871 tablet_service.cc:1467] Processing CreateTablet for tablet c95cd9642c1c46cabb23f6c62c6ad4f5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9eeb74ff004488ba3049c0e8e535bb5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.688344 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fa713b432740d98bcab918d251b54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.688787 26632 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:56.688897 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95cd9642c1c46cabb23f6c62c6ad4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.689831 26632 tablet_bootstrap.cc:654] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.689860 25478 tablet_service.cc:1467] Processing CreateTablet for tablet c95cd9642c1c46cabb23f6c62c6ad4f5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9eeb74ff004488ba3049c0e8e535bb5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.690147 25478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95cd9642c1c46cabb23f6c62c6ad4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.690647 26630 tablet_bootstrap.cc:492] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:56.691082 26632 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:56.691164 26632 ts_tablet_manager.cc:1367] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:56.691499 26630 tablet_bootstrap.cc:654] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.691613 26632 raft_consensus.cc:348] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.691736 26632 raft_consensus.cc:374] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.691776 26632 raft_consensus.cc:729] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.691947 26632 consensus_queue.cc:260] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.692122 26631 tablet_bootstrap.cc:492] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:56.692220 26632 ts_tablet_manager.cc:1398] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:56.693019 25476 tablet_service.cc:1467] Processing CreateTablet for tablet a1fa713b432740d98bcab918d251b54d (DEFAULT_TABLE table=rebalance_test_table_2 [id=f9eeb74ff004488ba3049c0e8e535bb5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:56.693065 26631 tablet_bootstrap.cc:654] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.693251 25476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fa713b432740d98bcab918d251b54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:56.694207 26631 tablet_bootstrap.cc:492] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:56.694301 26631 ts_tablet_manager.cc:1367] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:56.694732 26631 raft_consensus.cc:348] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.694830 26631 raft_consensus.cc:374] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.694852 26631 raft_consensus.cc:729] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.694981 26631 consensus_queue.cc:260] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.695242 26631 ts_tablet_manager.cc:1398] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:56.695384 26633 tablet_bootstrap.cc:492] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:56.695397 26631 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:56.696370 26633 tablet_bootstrap.cc:654] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.696416 26630 tablet_bootstrap.cc:492] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:56.696496 26630 ts_tablet_manager.cc:1367] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:58:56.696635 26631 tablet_bootstrap.cc:654] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.696992 26630 raft_consensus.cc:348] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.697098 26630 raft_consensus.cc:374] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.697134 26630 raft_consensus.cc:729] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.697274 26630 consensus_queue.cc:260] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.697580 26630 ts_tablet_manager.cc:1398] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:56.697762 26630 tablet_bootstrap.cc:492] T c95cd9642c1c46cabb23f6c62c6ad4f5 P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:56.697829 26631 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:56.697911 26631 ts_tablet_manager.cc:1367] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:58:56.698375 26631 raft_consensus.cc:348] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.698467 26631 raft_consensus.cc:374] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.698500 26631 raft_consensus.cc:729] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.698562 26630 tablet_bootstrap.cc:654] T c95cd9642c1c46cabb23f6c62c6ad4f5 P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.698632 26631 consensus_queue.cc:260] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.698923 26631 ts_tablet_manager.cc:1398] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:56.699070 26631 tablet_bootstrap.cc:492] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:56.699524 26633 tablet_bootstrap.cc:492] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:56.699625 26633 ts_tablet_manager.cc:1367] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:56.700074 26631 tablet_bootstrap.cc:654] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.700171 26633 raft_consensus.cc:348] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.700286 26633 raft_consensus.cc:374] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.700323 26633 raft_consensus.cc:729] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.700441 26633 consensus_queue.cc:260] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.701392 26633 ts_tablet_manager.cc:1398] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:56.701588 26633 tablet_bootstrap.cc:492] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:56.702383 26631 tablet_bootstrap.cc:492] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:56.702391 26630 tablet_bootstrap.cc:492] T c95cd9642c1c46cabb23f6c62c6ad4f5 P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:56.702473 26630 ts_tablet_manager.cc:1367] T c95cd9642c1c46cabb23f6c62c6ad4f5 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 07:58:56.702476 26631 ts_tablet_manager.cc:1367] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 07:58:56.702895 26630 raft_consensus.cc:348] T c95cd9642c1c46cabb23f6c62c6ad4f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.703008 26630 raft_consensus.cc:374] T c95cd9642c1c46cabb23f6c62c6ad4f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.702935 26631 raft_consensus.cc:348] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.703049 26630 raft_consensus.cc:729] T c95cd9642c1c46cabb23f6c62c6ad4f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.703065 26631 raft_consensus.cc:374] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.703104 26631 raft_consensus.cc:729] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.703150 26633 tablet_bootstrap.cc:654] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.703239 26630 consensus_queue.cc:260] T c95cd9642c1c46cabb23f6c62c6ad4f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.703248 26631 consensus_queue.cc:260] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.703554 26631 ts_tablet_manager.cc:1398] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 07:58:56.703646 26630 ts_tablet_manager.cc:1398] T c95cd9642c1c46cabb23f6c62c6ad4f5 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:56.703946 26630 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:56.704855 26633 tablet_bootstrap.cc:492] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:56.704880 26630 tablet_bootstrap.cc:654] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:56.704948 26633 ts_tablet_manager.cc:1367] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:56.705593 26633 raft_consensus.cc:348] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.705734 26633 raft_consensus.cc:374] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.705778 26633 raft_consensus.cc:729] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.705932 26633 consensus_queue.cc:260] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.706156 26630 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:56.706238 26630 ts_tablet_manager.cc:1367] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:56.706333 26633 ts_tablet_manager.cc:1398] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:56.707113 26630 raft_consensus.cc:348] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.707238 26630 raft_consensus.cc:374] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:56.707273 26630 raft_consensus.cc:729] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:56.707434 26630 consensus_queue.cc:260] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.707849 26630 ts_tablet_manager.cc:1398] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:56.731949 26544 raft_consensus.cc:482] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:56.732081 26544 raft_consensus.cc:504] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.732474 26505 raft_consensus.cc:482] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:56.732564 26544 leader_election.cc:288] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:56.732587 26505 raft_consensus.cc:504] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.732949 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056ba8d15d18409e946c905b8876097a" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:58:56.732969 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056ba8d15d18409e946c905b8876097a" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:56.733079 25629 raft_consensus.cc:2454] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:56.733096 25890 raft_consensus.cc:2454] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:56.733376 25434 leader_election.cc:302] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:56.733528 26544 raft_consensus.cc:2781] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:56.733592 26544 raft_consensus.cc:482] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:56.733639 26544 raft_consensus.cc:3037] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:56.733819 26505 leader_election.cc:288] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:56.734769 26544 raft_consensus.cc:504] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.735146 26544 leader_election.cc:288] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Requested vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:56.735479 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056ba8d15d18409e946c905b8876097a" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:58:56.735534 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056ba8d15d18409e946c905b8876097a" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:56.735596 25890 raft_consensus.cc:3037] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:56.735647 25629 raft_consensus.cc:3037] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:56.736582 25890 raft_consensus.cc:2454] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:56.736626 25629 raft_consensus.cc:2454] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:56.736970 25434 leader_election.cc:302] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:56.737190 26544 raft_consensus.cc:2781] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:56.737387 26544 raft_consensus.cc:686] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Becoming Leader. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Running, Role: LEADER
I20240502 07:58:56.737557 26544 consensus_queue.cc:237] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:56.738530 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056ba8d15d18409e946c905b8876097a" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" is_pre_election: true
I20240502 07:58:56.738788 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "056ba8d15d18409e946c905b8876097a" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:58:56.738962 25890 raft_consensus.cc:2379] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0dfa00c061c748b18772d5f0566944b5 in current term 1: Already voted for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in this term.
I20240502 07:58:56.739320 25564 leader_election.cc:302] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5; no voters: c3be11f0fe304e0f8e008dc1e4c41f70, e9596bfb8d324b4da93a4288771ed177
I20240502 07:58:56.739543 26505 raft_consensus.cc:2726] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 07:58:56.739570 25352 catalog_manager.cc:5484] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:56.866712 26647 raft_consensus.cc:482] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:56.866863 26647 raft_consensus.cc:504] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.867373 26647 leader_election.cc:288] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:58:56.867918 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fa713b432740d98bcab918d251b54d" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" is_pre_election: true
I20240502 07:58:56.867921 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fa713b432740d98bcab918d251b54d" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:58:56.868063 25759 raft_consensus.cc:2454] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:56.868081 25890 raft_consensus.cc:2454] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:56.868413 25433 leader_election.cc:302] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3be11f0fe304e0f8e008dc1e4c41f70, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:56.868595 26647 raft_consensus.cc:2781] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:56.868666 26647 raft_consensus.cc:482] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:56.868693 26647 raft_consensus.cc:3037] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:56.869613 26647 raft_consensus.cc:504] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.869962 26647 leader_election.cc:288] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Requested vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:58:56.870348 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fa713b432740d98bcab918d251b54d" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:58:56.870344 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1fa713b432740d98bcab918d251b54d" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
I20240502 07:58:56.870460 25890 raft_consensus.cc:3037] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:56.870467 25759 raft_consensus.cc:3037] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:56.871352 25890 raft_consensus.cc:2454] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:56.871428 25759 raft_consensus.cc:2454] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:56.871675 25433 leader_election.cc:302] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3be11f0fe304e0f8e008dc1e4c41f70, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:56.871872 26647 raft_consensus.cc:2781] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:56.871999 26647 raft_consensus.cc:686] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Becoming Leader. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Running, Role: LEADER
I20240502 07:58:56.872159 26647 consensus_queue.cc:237] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:56.874162 25352 catalog_manager.cc:5484] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:57.075845 26510 raft_consensus.cc:482] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:57.075997 26510 raft_consensus.cc:504] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:57.077132 26510 leader_election.cc:288] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:57.081635 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95cd9642c1c46cabb23f6c62c6ad4f5" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:57.081722 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95cd9642c1c46cabb23f6c62c6ad4f5" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" is_pre_election: true
I20240502 07:58:57.081820 25629 raft_consensus.cc:2454] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 0.
I20240502 07:58:57.081861 25497 raft_consensus.cc:2454] T c95cd9642c1c46cabb23f6c62c6ad4f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 0.
I20240502 07:58:57.082175 25827 leader_election.cc:302] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:57.082422 26510 raft_consensus.cc:2781] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:57.082486 26510 raft_consensus.cc:482] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:57.082525 26510 raft_consensus.cc:3037] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:57.083458 26510 raft_consensus.cc:504] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:57.083846 26510 leader_election.cc:288] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 election: Requested vote from peers c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:57.084159 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95cd9642c1c46cabb23f6c62c6ad4f5" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
I20240502 07:58:57.084295 25497 raft_consensus.cc:3037] T c95cd9642c1c46cabb23f6c62c6ad4f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:57.084316 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95cd9642c1c46cabb23f6c62c6ad4f5" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:57.084434 25629 raft_consensus.cc:3037] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:57.085345 25497 raft_consensus.cc:2454] T c95cd9642c1c46cabb23f6c62c6ad4f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 1.
I20240502 07:58:57.085345 25629 raft_consensus.cc:2454] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 1.
I20240502 07:58:57.085656 25827 leader_election.cc:302] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:57.085834 26510 raft_consensus.cc:2781] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:57.086052 26510 raft_consensus.cc:686] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Becoming Leader. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Running, Role: LEADER
I20240502 07:58:57.086381 26510 consensus_queue.cc:237] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:57.088793 25352 catalog_manager.cc:5484] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9596bfb8d324b4da93a4288771ed177 (127.18.13.132). New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:57.266037 26655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:57.266382 26655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 07:58:57.266911 26647 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 07:58:57.270918 26655 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:58:57.271301 26647 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240502 07:58:57.311213 26544 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:57.317986 26647 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:57.372995 25629 raft_consensus.cc:1261] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:57.373946 26510 consensus_queue.cc:1035] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:57.374629 25497 raft_consensus.cc:1261] T c95cd9642c1c46cabb23f6c62c6ad4f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:57.376062 26510 consensus_queue.cc:1035] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 114.411 ms
time per row: 0.0572053 ms
I20240502 07:58:57.433235 25355 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40498:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:57.438897 25740 tablet_service.cc:1467] Processing CreateTablet for tablet 03016ecfa11d4987a2e9d2368a66a5a2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c7fb9171c9548a7ad2d45e295d780cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:57.439282 25740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03016ecfa11d4987a2e9d2368a66a5a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:57.439514 25476 tablet_service.cc:1467] Processing CreateTablet for tablet 03016ecfa11d4987a2e9d2368a66a5a2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c7fb9171c9548a7ad2d45e295d780cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:57.439690 25609 tablet_service.cc:1467] Processing CreateTablet for tablet 03016ecfa11d4987a2e9d2368a66a5a2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c7fb9171c9548a7ad2d45e295d780cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:57.439848 25476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03016ecfa11d4987a2e9d2368a66a5a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:57.439978 25609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03016ecfa11d4987a2e9d2368a66a5a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:57.442034 26716 tablet_bootstrap.cc:492] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:57.442209 25478 tablet_service.cc:1467] Processing CreateTablet for tablet fd68090fdadf45faa0e3b2936b30047e (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c7fb9171c9548a7ad2d45e295d780cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:57.442318 26715 tablet_bootstrap.cc:492] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:57.442451 25478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd68090fdadf45faa0e3b2936b30047e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:57.442945 26716 tablet_bootstrap.cc:654] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:57.443197 26715 tablet_bootstrap.cc:654] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:57.443698 26717 tablet_bootstrap.cc:492] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:57.444205 26715 tablet_bootstrap.cc:492] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:57.444281 26715 ts_tablet_manager.cc:1367] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:57.444581 26717 tablet_bootstrap.cc:654] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:57.444737 26715 raft_consensus.cc:348] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:57.444835 26715 raft_consensus.cc:374] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:57.444864 26715 raft_consensus.cc:729] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:57.444876 25740 tablet_service.cc:1467] Processing CreateTablet for tablet fd68090fdadf45faa0e3b2936b30047e (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c7fb9171c9548a7ad2d45e295d780cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:57.444881 25738 tablet_service.cc:1467] Processing CreateTablet for tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c7fb9171c9548a7ad2d45e295d780cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:57.445004 26715 consensus_queue.cc:260] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:57.445156 25740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd68090fdadf45faa0e3b2936b30047e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:57.445286 26715 ts_tablet_manager.cc:1398] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:57.446128 25609 tablet_service.cc:1467] Processing CreateTablet for tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c7fb9171c9548a7ad2d45e295d780cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:57.446415 25609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:57.447103 25738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:57.447245 25871 tablet_service.cc:1467] Processing CreateTablet for tablet fd68090fdadf45faa0e3b2936b30047e (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c7fb9171c9548a7ad2d45e295d780cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:57.447506 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd68090fdadf45faa0e3b2936b30047e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:57.448041 25870 tablet_service.cc:1467] Processing CreateTablet for tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=7c7fb9171c9548a7ad2d45e295d780cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:57.448588 26715 tablet_bootstrap.cc:492] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:57.449143 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:57.449487 26715 tablet_bootstrap.cc:654] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:57.450554 26715 tablet_bootstrap.cc:492] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:57.450556 26716 tablet_bootstrap.cc:492] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:57.450629 26716 ts_tablet_manager.cc:1367] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 07:58:57.450628 26715 ts_tablet_manager.cc:1367] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:57.451051 26716 raft_consensus.cc:348] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:57.451047 26715 raft_consensus.cc:348] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.451148 26715 raft_consensus.cc:374] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:57.451148 26716 raft_consensus.cc:374] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:57.451187 26715 raft_consensus.cc:729] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:57.451187 26716 raft_consensus.cc:729] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:57.451310 26716 consensus_queue.cc:260] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:57.451310 26715 consensus_queue.cc:260] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.451586 26715 ts_tablet_manager.cc:1398] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:57.451753 26715 tablet_bootstrap.cc:492] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:57.451938 26716 ts_tablet_manager.cc:1398] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:57.452426 26716 tablet_bootstrap.cc:492] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:57.452546 26715 tablet_bootstrap.cc:654] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:57.453367 26722 tablet_bootstrap.cc:492] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:57.453884 26716 tablet_bootstrap.cc:654] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:57.454501 26717 tablet_bootstrap.cc:492] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:57.454916 26717 ts_tablet_manager.cc:1367] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 07:58:57.455451 26717 raft_consensus.cc:348] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:57.455585 26717 raft_consensus.cc:374] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:57.455616 26717 raft_consensus.cc:729] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:57.455790 26717 consensus_queue.cc:260] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:57.456741 26717 ts_tablet_manager.cc:1398] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:57.457362 26715 tablet_bootstrap.cc:492] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:57.457448 26715 ts_tablet_manager.cc:1367] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240502 07:58:57.457841 26715 raft_consensus.cc:348] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.457939 26715 raft_consensus.cc:374] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:57.457969 26715 raft_consensus.cc:729] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:57.457963 26717 tablet_bootstrap.cc:492] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:57.458070 26715 consensus_queue.cc:260] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.458293 26715 ts_tablet_manager.cc:1398] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:57.459146 26717 tablet_bootstrap.cc:654] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:57.460450 26716 tablet_bootstrap.cc:492] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:57.460508 26722 tablet_bootstrap.cc:654] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:57.460551 26716 ts_tablet_manager.cc:1367] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 07:58:57.461053 26716 raft_consensus.cc:348] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.461167 26716 raft_consensus.cc:374] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:57.461169 26717 tablet_bootstrap.cc:492] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:57.461195 26716 raft_consensus.cc:729] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:57.461278 26717 ts_tablet_manager.cc:1367] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:57.461323 26716 consensus_queue.cc:260] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.461640 26716 ts_tablet_manager.cc:1398] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:57.461690 26717 raft_consensus.cc:348] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.461810 26717 raft_consensus.cc:374] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:57.461848 26717 raft_consensus.cc:729] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:57.461939 26722 tablet_bootstrap.cc:492] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:57.461984 26717 consensus_queue.cc:260] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.462033 26722 ts_tablet_manager.cc:1367] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 07:58:57.462263 26717 ts_tablet_manager.cc:1398] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:57.462749 26722 raft_consensus.cc:348] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.462869 26722 raft_consensus.cc:374] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:57.462910 26722 raft_consensus.cc:729] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:57.463088 26722 consensus_queue.cc:260] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.463500 26722 ts_tablet_manager.cc:1398] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:57.463665 26722 tablet_bootstrap.cc:492] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:57.463735 26548 raft_consensus.cc:482] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:57.463850 26548 raft_consensus.cc:504] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.464267 26548 leader_election.cc:288] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:57.464620 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5958bb5c8fc4c1d91ae8f1e8b3861d0" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:57.464632 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5958bb5c8fc4c1d91ae8f1e8b3861d0" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:58:57.464740 25890 raft_consensus.cc:2454] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:57.464745 25629 raft_consensus.cc:2454] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:57.465175 25695 leader_election.cc:302] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:57.465497 26548 raft_consensus.cc:2781] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:57.465557 26548 raft_consensus.cc:482] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:57.465602 26548 raft_consensus.cc:3037] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:57.466475 26548 raft_consensus.cc:504] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.466823 26548 leader_election.cc:288] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Requested vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:57.467111 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5958bb5c8fc4c1d91ae8f1e8b3861d0" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:58:57.467130 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5958bb5c8fc4c1d91ae8f1e8b3861d0" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:57.467221 25890 raft_consensus.cc:3037] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:57.467237 25629 raft_consensus.cc:3037] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:57.467446 26722 tablet_bootstrap.cc:654] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:57.468050 25890 raft_consensus.cc:2454] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:58:57.468232 25629 raft_consensus.cc:2454] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:58:57.468315 25695 leader_election.cc:302] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:57.468647 26548 raft_consensus.cc:2781] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:57.468858 26548 raft_consensus.cc:686] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Becoming Leader. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Running, Role: LEADER
I20240502 07:58:57.468982 26722 tablet_bootstrap.cc:492] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:57.469069 26722 ts_tablet_manager.cc:1367] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 07:58:57.469017 26548 consensus_queue.cc:237] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.469509 26722 raft_consensus.cc:348] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.469627 26722 raft_consensus.cc:374] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:57.469674 26722 raft_consensus.cc:729] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:57.469798 26722 consensus_queue.cc:260] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.470067 26722 ts_tablet_manager.cc:1398] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:57.470950 25355 catalog_manager.cc:5484] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: term changed from 0 to 1, leader changed from <none> to 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131). New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:57.566704 26709 raft_consensus.cc:482] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:57.566881 26709 raft_consensus.cc:504] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.567471 26709 leader_election.cc:288] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:57.567867 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd68090fdadf45faa0e3b2936b30047e" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" is_pre_election: true
I20240502 07:58:57.567902 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd68090fdadf45faa0e3b2936b30047e" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:58:57.568028 25759 raft_consensus.cc:2454] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:57.568039 25890 raft_consensus.cc:2454] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:57.568378 25433 leader_election.cc:302] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3be11f0fe304e0f8e008dc1e4c41f70, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:57.568568 26709 raft_consensus.cc:2781] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:57.568627 26709 raft_consensus.cc:482] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:57.568655 26709 raft_consensus.cc:3037] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:57.569550 26709 raft_consensus.cc:504] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.569882 26709 leader_election.cc:288] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Requested vote from peers 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:57.570263 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd68090fdadf45faa0e3b2936b30047e" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:58:57.570263 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd68090fdadf45faa0e3b2936b30047e" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
I20240502 07:58:57.570395 25890 raft_consensus.cc:3037] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:57.570395 25759 raft_consensus.cc:3037] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:57.571367 25890 raft_consensus.cc:2454] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:57.571367 25759 raft_consensus.cc:2454] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:57.571702 25432 leader_election.cc:302] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:57.571900 26709 raft_consensus.cc:2781] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:57.571990 26709 raft_consensus.cc:686] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Becoming Leader. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Running, Role: LEADER
I20240502 07:58:57.572140 26709 consensus_queue.cc:237] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:57.573871 25355 catalog_manager.cc:5484] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:57.765558 26548 raft_consensus.cc:482] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:57.765697 26548 raft_consensus.cc:504] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:57.766137 26548 leader_election.cc:288] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:57.766669 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03016ecfa11d4987a2e9d2368a66a5a2" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" is_pre_election: true
I20240502 07:58:57.766705 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03016ecfa11d4987a2e9d2368a66a5a2" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:57.766821 25497 raft_consensus.cc:2454] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:57.766844 25629 raft_consensus.cc:2454] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:57.767179 25696 leader_election.cc:302] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, 14d8fd1fc3a04d9d967b8033df84f34a; no voters:
I20240502 07:58:57.767370 26548 raft_consensus.cc:2781] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:57.767431 26548 raft_consensus.cc:482] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:57.767463 26548 raft_consensus.cc:3037] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:57.768361 26548 raft_consensus.cc:504] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:57.768741 26548 leader_election.cc:288] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Requested vote from peers c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:57.769165 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03016ecfa11d4987a2e9d2368a66a5a2" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
I20240502 07:58:57.769232 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03016ecfa11d4987a2e9d2368a66a5a2" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:57.769289 25497 raft_consensus.cc:3037] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:57.769369 25629 raft_consensus.cc:3037] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:57.770129 25629 raft_consensus.cc:2454] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:58:57.770138 25497 raft_consensus.cc:2454] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:58:57.770429 25696 leader_election.cc:302] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:57.770583 26548 raft_consensus.cc:2781] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:57.770679 26548 raft_consensus.cc:686] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Becoming Leader. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Running, Role: LEADER
I20240502 07:58:57.770844 26548 consensus_queue.cc:237] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:57.772651 25355 catalog_manager.cc:5484] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: term changed from 0 to 1, leader changed from <none> to 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131). New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:57.933318 26732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:57.933588 26732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:57.936853 26732 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 07:58:57.963310 26548 consensus_queue.cc:1035] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:57.965958 26548 consensus_queue.cc:1035] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:58.017290 26709 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:58.022941 26693 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:58.025269 25629 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:58.025524 25497 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:58.026002 26548 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:58.026335 26548 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 114.069 ms
time per row: 0.0570345 ms
I20240502 07:58:58.098862 25354 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40498:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:58.105062 25609 tablet_service.cc:1467] Processing CreateTablet for tablet 8904fa47fc274c14acd291b4c91964be (DEFAULT_TABLE table=rebalance_test_table_4 [id=f72d1ad6e00b46cf99f485a9ef60b9b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.105181 25478 tablet_service.cc:1467] Processing CreateTablet for tablet 8904fa47fc274c14acd291b4c91964be (DEFAULT_TABLE table=rebalance_test_table_4 [id=f72d1ad6e00b46cf99f485a9ef60b9b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.105430 25609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8904fa47fc274c14acd291b4c91964be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.105515 25478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8904fa47fc274c14acd291b4c91964be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.106005 25740 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa60608089c4f50bcee339d01815a65 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f72d1ad6e00b46cf99f485a9ef60b9b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.106027 25608 tablet_service.cc:1467] Processing CreateTablet for tablet 16a130f60be74f568cc53a70692a6838 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f72d1ad6e00b46cf99f485a9ef60b9b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.106245 25608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a130f60be74f568cc53a70692a6838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.106249 25740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa60608089c4f50bcee339d01815a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.107934 25738 tablet_service.cc:1467] Processing CreateTablet for tablet 16a130f60be74f568cc53a70692a6838 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f72d1ad6e00b46cf99f485a9ef60b9b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.108165 25738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a130f60be74f568cc53a70692a6838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.108937 26787 tablet_bootstrap.cc:492] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:58.108935 25476 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa60608089c4f50bcee339d01815a65 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f72d1ad6e00b46cf99f485a9ef60b9b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.109191 25476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa60608089c4f50bcee339d01815a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.109958 26787 tablet_bootstrap.cc:654] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.110495 26786 tablet_bootstrap.cc:492] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:58.110675 26785 tablet_bootstrap.cc:492] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:58.111356 26787 tablet_bootstrap.cc:492] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:58.111438 26787 ts_tablet_manager.cc:1367] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:58.111466 25870 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa60608089c4f50bcee339d01815a65 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f72d1ad6e00b46cf99f485a9ef60b9b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.111753 26786 tablet_bootstrap.cc:654] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.111766 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa60608089c4f50bcee339d01815a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.112020 26787 raft_consensus.cc:348] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:58.112145 26787 raft_consensus.cc:374] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.112192 26787 raft_consensus.cc:729] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.112341 26787 consensus_queue.cc:260] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:58.112464 26548 raft_consensus.cc:482] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:58.112540 26548 raft_consensus.cc:504] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:58.112931 26548 leader_election.cc:288] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:58:58.113008 26786 tablet_bootstrap.cc:492] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:58.113094 26786 ts_tablet_manager.cc:1367] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:58:58.113269 26787 ts_tablet_manager.cc:1398] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:58.113428 26787 tablet_bootstrap.cc:492] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:58.113529 26786 raft_consensus.cc:348] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.113651 26786 raft_consensus.cc:374] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.113685 26786 raft_consensus.cc:729] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
W20240502 07:58:58.113680 25695 leader_election.cc:341] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865): Not found: Tablet not found: 0aa60608089c4f50bcee339d01815a65
I20240502 07:58:58.113799 26786 consensus_queue.cc:260] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.114076 26786 ts_tablet_manager.cc:1398] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:58.114236 26786 tablet_bootstrap.cc:492] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:58.114312 26787 tablet_bootstrap.cc:654] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.115103 26786 tablet_bootstrap.cc:654] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.115839 26785 tablet_bootstrap.cc:654] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.111466 25871 tablet_service.cc:1467] Processing CreateTablet for tablet 8904fa47fc274c14acd291b4c91964be (DEFAULT_TABLE table=rebalance_test_table_4 [id=f72d1ad6e00b46cf99f485a9ef60b9b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.116062 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8904fa47fc274c14acd291b4c91964be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.116765 26787 tablet_bootstrap.cc:492] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:58.116876 26787 ts_tablet_manager.cc:1367] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:58.117246 26787 raft_consensus.cc:348] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:58.117342 26787 raft_consensus.cc:374] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.117377 26787 raft_consensus.cc:729] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.117522 26787 consensus_queue.cc:260] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:58.117816 26787 ts_tablet_manager.cc:1398] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:58.118258 26785 tablet_bootstrap.cc:492] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:58.118351 26785 ts_tablet_manager.cc:1367] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240502 07:58:58.118764 26785 raft_consensus.cc:348] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.118885 26785 raft_consensus.cc:374] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.118932 26785 raft_consensus.cc:729] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.119060 26785 consensus_queue.cc:260] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.119282 26785 ts_tablet_manager.cc:1398] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:58.119455 26785 tablet_bootstrap.cc:492] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:58.120478 26785 tablet_bootstrap.cc:654] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.121084 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa60608089c4f50bcee339d01815a65" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" is_pre_election: true
W20240502 07:58:58.121675 25696 leader_election.cc:341] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377): Illegal state: must be running to vote when last-logged opid is not known
I20240502 07:58:58.121783 25696 leader_election.cc:302] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a; no voters: c3be11f0fe304e0f8e008dc1e4c41f70, e9596bfb8d324b4da93a4288771ed177
I20240502 07:58:58.121915 26785 tablet_bootstrap.cc:492] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:58.122009 26785 ts_tablet_manager.cc:1367] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:58.122017 26548 raft_consensus.cc:2726] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 07:58:58.122412 25478 tablet_service.cc:1467] Processing CreateTablet for tablet 16a130f60be74f568cc53a70692a6838 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f72d1ad6e00b46cf99f485a9ef60b9b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.122474 26785 raft_consensus.cc:348] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:58.122583 26785 raft_consensus.cc:374] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.122622 26785 raft_consensus.cc:729] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.122687 25478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a130f60be74f568cc53a70692a6838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.122764 26785 consensus_queue.cc:260] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:58.123009 26785 ts_tablet_manager.cc:1398] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:58.123389 26790 tablet_bootstrap.cc:492] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:58.124395 26790 tablet_bootstrap.cc:654] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.124493 26786 tablet_bootstrap.cc:492] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:58.124581 26786 ts_tablet_manager.cc:1367] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20240502 07:58:58.125146 26786 raft_consensus.cc:348] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:58.125274 26786 raft_consensus.cc:374] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.125319 26786 raft_consensus.cc:729] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.125478 26790 tablet_bootstrap.cc:492] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:58.125447 26786 consensus_queue.cc:260] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:58.125568 26790 ts_tablet_manager.cc:1367] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:58.125752 26786 ts_tablet_manager.cc:1398] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:58.125928 26786 tablet_bootstrap.cc:492] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:58.125988 26790 raft_consensus.cc:348] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:58.126103 26790 raft_consensus.cc:374] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.126142 26790 raft_consensus.cc:729] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.126296 26790 consensus_queue.cc:260] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:58.126571 26790 ts_tablet_manager.cc:1398] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:58.126754 26790 tablet_bootstrap.cc:492] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:58.126909 26786 tablet_bootstrap.cc:654] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.127574 26790 tablet_bootstrap.cc:654] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.128131 26786 tablet_bootstrap.cc:492] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:58.128221 26786 ts_tablet_manager.cc:1367] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 07:58:58.128698 26790 tablet_bootstrap.cc:492] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:58.128706 26786 raft_consensus.cc:348] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:58.128789 26790 ts_tablet_manager.cc:1367] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:58.128836 26786 raft_consensus.cc:374] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.128878 26786 raft_consensus.cc:729] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.129009 26786 consensus_queue.cc:260] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:58.129225 26790 raft_consensus.cc:348] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.129287 26786 ts_tablet_manager.cc:1398] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:58.129333 26790 raft_consensus.cc:374] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.129366 26790 raft_consensus.cc:729] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.129477 26790 consensus_queue.cc:260] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.129753 26790 ts_tablet_manager.cc:1398] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:58.144584 26712 raft_consensus.cc:482] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:58.144749 26712 raft_consensus.cc:504] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:58.145222 26712 leader_election.cc:288] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205), c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:58:58.145762 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa60608089c4f50bcee339d01815a65" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" is_pre_election: true
I20240502 07:58:58.145908 25497 raft_consensus.cc:2454] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 0.
I20240502 07:58:58.146241 25827 leader_election.cc:302] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3be11f0fe304e0f8e008dc1e4c41f70, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:58.146423 26712 raft_consensus.cc:2781] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:58.146485 26712 raft_consensus.cc:482] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:58.146538 26712 raft_consensus.cc:3037] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.147432 26712 raft_consensus.cc:504] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:58.147814 26712 leader_election.cc:288] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 election: Requested vote from peers 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205), c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:58:58.148156 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa60608089c4f50bcee339d01815a65" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
I20240502 07:58:58.148276 25497 raft_consensus.cc:3037] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.149410 25497 raft_consensus.cc:2454] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 1.
I20240502 07:58:58.149462 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa60608089c4f50bcee339d01815a65" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" is_pre_election: true
I20240502 07:58:58.149504 25760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa60608089c4f50bcee339d01815a65" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
I20240502 07:58:58.149595 25759 raft_consensus.cc:2454] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 0.
I20240502 07:58:58.149718 25827 leader_election.cc:302] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3be11f0fe304e0f8e008dc1e4c41f70, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:58.149906 26712 raft_consensus.cc:2781] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:58.150120 26712 raft_consensus.cc:686] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Becoming Leader. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Running, Role: LEADER
I20240502 07:58:58.150352 26712 consensus_queue.cc:237] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:58.152206 25355 catalog_manager.cc:5484] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9596bfb8d324b4da93a4288771ed177 (127.18.13.132). New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:58.191285 26693 raft_consensus.cc:482] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:58.191453 26693 raft_consensus.cc:504] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.191962 26693 leader_election.cc:288] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:58.192412 25628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8904fa47fc274c14acd291b4c91964be" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:58.192546 25628 raft_consensus.cc:2454] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:58.192534 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8904fa47fc274c14acd291b4c91964be" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:58:58.192723 25891 raft_consensus.cc:2454] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:58.192900 25434 leader_election.cc:302] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:58.193078 26693 raft_consensus.cc:2781] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:58.193140 26693 raft_consensus.cc:482] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:58.193181 26693 raft_consensus.cc:3037] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.194037 26693 raft_consensus.cc:504] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.194365 26693 leader_election.cc:288] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Requested vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:58.194710 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8904fa47fc274c14acd291b4c91964be" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:58:58.194716 25628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8904fa47fc274c14acd291b4c91964be" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:58.194823 25891 raft_consensus.cc:3037] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.194844 25628 raft_consensus.cc:3037] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.195601 25628 raft_consensus.cc:2454] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:58.195605 25891 raft_consensus.cc:2454] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:58.195901 25434 leader_election.cc:302] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:58.196055 26693 raft_consensus.cc:2781] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:58.196141 26693 raft_consensus.cc:686] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Becoming Leader. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Running, Role: LEADER
I20240502 07:58:58.196290 26693 consensus_queue.cc:237] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.198102 25355 catalog_manager.cc:5484] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:58.205605 26693 raft_consensus.cc:482] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:58.205751 26693 raft_consensus.cc:504] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:58.206151 26693 leader_election.cc:288] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:58.206600 25628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a130f60be74f568cc53a70692a6838" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:58.206669 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a130f60be74f568cc53a70692a6838" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" is_pre_election: true
I20240502 07:58:58.206771 25628 raft_consensus.cc:2454] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:58.206837 25759 raft_consensus.cc:2454] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:58.207088 25434 leader_election.cc:302] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:58.207330 26693 raft_consensus.cc:2781] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:58.207376 26693 raft_consensus.cc:482] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:58.207418 26693 raft_consensus.cc:3037] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.208361 26693 raft_consensus.cc:504] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:58.208705 26693 leader_election.cc:288] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Requested vote from peers 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:58.209069 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a130f60be74f568cc53a70692a6838" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
I20240502 07:58:58.209125 25628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16a130f60be74f568cc53a70692a6838" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:58.209211 25759 raft_consensus.cc:3037] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.209249 25628 raft_consensus.cc:3037] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.210062 25759 raft_consensus.cc:2454] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:58.210062 25628 raft_consensus.cc:2454] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:58.210366 25434 leader_election.cc:302] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:58.210525 26693 raft_consensus.cc:2781] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:58.210644 26693 raft_consensus.cc:686] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Becoming Leader. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Running, Role: LEADER
I20240502 07:58:58.210839 26693 consensus_queue.cc:237] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:58.212527 25355 catalog_manager.cc:5484] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:58.334669 26803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:58.334926 26803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:58.338075 26803 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 07:58:58.420994 25628 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:58.421437 25891 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:58.422204 26693 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:58.427896 26709 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:58.432631 25497 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:58.433976 26799 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:58.435241 25759 raft_consensus.cc:1261] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:58.435969 25628 raft_consensus.cc:1261] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:58.436389 26693 consensus_queue.cc:1035] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:58.442335 26693 consensus_queue.cc:1035] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:58.447477 25760 raft_consensus.cc:3037] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.448707 25760 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:58.449223 26799 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 120.486 ms
time per row: 0.0602428 ms
I20240502 07:58:58.504143 25355 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40498:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:58.509781 25608 tablet_service.cc:1467] Processing CreateTablet for tablet a3bc416015dc40d88743215d2cf46230 (DEFAULT_TABLE table=rebalance_test_table_5 [id=a5520c28ad46406f9fdf6855d5dfa37b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.509958 25871 tablet_service.cc:1467] Processing CreateTablet for tablet a3bc416015dc40d88743215d2cf46230 (DEFAULT_TABLE table=rebalance_test_table_5 [id=a5520c28ad46406f9fdf6855d5dfa37b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.510129 25608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bc416015dc40d88743215d2cf46230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.510289 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bc416015dc40d88743215d2cf46230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.510761 25738 tablet_service.cc:1467] Processing CreateTablet for tablet a3bc416015dc40d88743215d2cf46230 (DEFAULT_TABLE table=rebalance_test_table_5 [id=a5520c28ad46406f9fdf6855d5dfa37b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.510792 25740 tablet_service.cc:1467] Processing CreateTablet for tablet fc1ac05112724e0ea1f84f648598304e (DEFAULT_TABLE table=rebalance_test_table_5 [id=a5520c28ad46406f9fdf6855d5dfa37b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.511073 25738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bc416015dc40d88743215d2cf46230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.512239 26785 tablet_bootstrap.cc:492] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:58.512596 26790 tablet_bootstrap.cc:492] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:58.513017 26785 tablet_bootstrap.cc:654] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.513252 26787 tablet_bootstrap.cc:492] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:58.513397 25740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1ac05112724e0ea1f84f648598304e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.513428 26790 tablet_bootstrap.cc:654] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.514266 26787 tablet_bootstrap.cc:654] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.514472 25870 tablet_service.cc:1467] Processing CreateTablet for tablet d6929d2560404acb9bb53f54a181f958 (DEFAULT_TABLE table=rebalance_test_table_5 [id=a5520c28ad46406f9fdf6855d5dfa37b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.514566 26790 tablet_bootstrap.cc:492] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:58.514649 26790 ts_tablet_manager.cc:1367] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:58.514701 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6929d2560404acb9bb53f54a181f958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.515054 26790 raft_consensus.cc:348] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.515138 26790 raft_consensus.cc:374] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.515158 26790 raft_consensus.cc:729] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.515271 26790 consensus_queue.cc:260] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.515520 26790 ts_tablet_manager.cc:1398] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:58.516137 25609 tablet_service.cc:1467] Processing CreateTablet for tablet d6929d2560404acb9bb53f54a181f958 (DEFAULT_TABLE table=rebalance_test_table_5 [id=a5520c28ad46406f9fdf6855d5dfa37b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.516230 25478 tablet_service.cc:1467] Processing CreateTablet for tablet d6929d2560404acb9bb53f54a181f958 (DEFAULT_TABLE table=rebalance_test_table_5 [id=a5520c28ad46406f9fdf6855d5dfa37b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.516387 25609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6929d2560404acb9bb53f54a181f958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.516492 25478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6929d2560404acb9bb53f54a181f958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.516847 26790 tablet_bootstrap.cc:492] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:58.517732 26790 tablet_bootstrap.cc:654] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.518710 25871 tablet_service.cc:1467] Processing CreateTablet for tablet fc1ac05112724e0ea1f84f648598304e (DEFAULT_TABLE table=rebalance_test_table_5 [id=a5520c28ad46406f9fdf6855d5dfa37b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.518942 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1ac05112724e0ea1f84f648598304e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.518922 25608 tablet_service.cc:1467] Processing CreateTablet for tablet fc1ac05112724e0ea1f84f648598304e (DEFAULT_TABLE table=rebalance_test_table_5 [id=a5520c28ad46406f9fdf6855d5dfa37b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:58.519155 25608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1ac05112724e0ea1f84f648598304e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:58.519658 26786 tablet_bootstrap.cc:492] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:58.520707 26786 tablet_bootstrap.cc:654] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.521185 26787 tablet_bootstrap.cc:492] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:58.521273 26787 ts_tablet_manager.cc:1367] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 07:58:58.521823 26787 raft_consensus.cc:348] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.521945 26787 raft_consensus.cc:374] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.521978 26787 raft_consensus.cc:729] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.521977 26786 tablet_bootstrap.cc:492] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:58.522058 26786 ts_tablet_manager.cc:1367] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:58.522145 26790 tablet_bootstrap.cc:492] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:58.522228 26790 ts_tablet_manager.cc:1367] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 07:58:58.522498 26786 raft_consensus.cc:348] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.522598 26786 raft_consensus.cc:374] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.522635 26786 raft_consensus.cc:729] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.522603 26790 raft_consensus.cc:348] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.522692 26790 raft_consensus.cc:374] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.522727 26790 raft_consensus.cc:729] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.522769 26786 consensus_queue.cc:260] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.522831 26790 consensus_queue.cc:260] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.523041 26786 ts_tablet_manager.cc:1398] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:58.523088 26790 ts_tablet_manager.cc:1398] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:58.523229 26790 tablet_bootstrap.cc:492] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:58.523665 26787 consensus_queue.cc:260] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.524245 26785 tablet_bootstrap.cc:492] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:58.524339 26785 ts_tablet_manager.cc:1367] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20240502 07:58:58.524443 26790 tablet_bootstrap.cc:654] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.524847 26785 raft_consensus.cc:348] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.524961 26785 raft_consensus.cc:374] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.524994 26785 raft_consensus.cc:729] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.525115 26785 consensus_queue.cc:260] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.525396 26785 ts_tablet_manager.cc:1398] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:58.525372 26787 ts_tablet_manager.cc:1398] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:58.525588 26785 tablet_bootstrap.cc:492] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:58.525813 26790 tablet_bootstrap.cc:492] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:58.525885 26790 ts_tablet_manager.cc:1367] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:58.526350 26790 raft_consensus.cc:348] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.526444 26790 raft_consensus.cc:374] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.526473 26790 raft_consensus.cc:729] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.526535 26785 tablet_bootstrap.cc:654] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.526597 26790 consensus_queue.cc:260] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.526969 26790 ts_tablet_manager.cc:1398] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:58.528106 26787 tablet_bootstrap.cc:492] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:58.529034 26787 tablet_bootstrap.cc:654] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.529812 26785 tablet_bootstrap.cc:492] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:58.529902 26785 ts_tablet_manager.cc:1367] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 07:58:58.530398 26785 raft_consensus.cc:348] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.530619 26785 raft_consensus.cc:374] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.530670 26785 raft_consensus.cc:729] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.530836 26785 consensus_queue.cc:260] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.531144 26785 ts_tablet_manager.cc:1398] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 07:58:58.531340 26785 tablet_bootstrap.cc:492] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:58.531330 26787 tablet_bootstrap.cc:492] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:58.531415 26787 ts_tablet_manager.cc:1367] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:58.532159 26785 tablet_bootstrap.cc:654] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:58.532470 26787 raft_consensus.cc:348] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.532588 26787 raft_consensus.cc:374] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.532629 26787 raft_consensus.cc:729] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.532770 26787 consensus_queue.cc:260] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.533051 26787 ts_tablet_manager.cc:1398] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:58.533265 26785 tablet_bootstrap.cc:492] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:58.533324 26785 ts_tablet_manager.cc:1367] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:58:58.533624 26785 raft_consensus.cc:348] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.533736 26785 raft_consensus.cc:374] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:58.533774 26785 raft_consensus.cc:729] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:58.533890 26785 consensus_queue.cc:260] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.534205 26785 ts_tablet_manager.cc:1398] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:58.537320 26728 raft_consensus.cc:482] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:58.537401 26728 raft_consensus.cc:504] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.537781 26728 leader_election.cc:288] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:58.538053 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bc416015dc40d88743215d2cf46230" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:58.538182 25629 raft_consensus.cc:2454] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:58.538167 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bc416015dc40d88743215d2cf46230" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:58:58.538311 25891 raft_consensus.cc:2454] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:58.538455 25696 leader_election.cc:302] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, 14d8fd1fc3a04d9d967b8033df84f34a; no voters:
I20240502 07:58:58.538610 26728 raft_consensus.cc:2781] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:58.538669 26728 raft_consensus.cc:482] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:58.538712 26728 raft_consensus.cc:3037] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.539637 26728 raft_consensus.cc:504] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.540004 26728 leader_election.cc:288] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Requested vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:58.540306 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bc416015dc40d88743215d2cf46230" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:58.540307 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3bc416015dc40d88743215d2cf46230" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:58:58.540424 25629 raft_consensus.cc:3037] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.540429 25891 raft_consensus.cc:3037] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.541348 25891 raft_consensus.cc:2454] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:58:58.541348 25629 raft_consensus.cc:2454] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:58:58.541626 25696 leader_election.cc:302] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, 14d8fd1fc3a04d9d967b8033df84f34a; no voters:
I20240502 07:58:58.541798 26728 raft_consensus.cc:2781] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:58.541886 26728 raft_consensus.cc:686] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Becoming Leader. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Running, Role: LEADER
I20240502 07:58:58.542021 26728 consensus_queue.cc:237] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.543553 25349 catalog_manager.cc:5484] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: term changed from 0 to 1, leader changed from <none> to 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131). New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:58.658526 26799 raft_consensus.cc:482] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:58.658653 26799 raft_consensus.cc:504] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.659093 26799 leader_election.cc:288] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:58:58.659381 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6929d2560404acb9bb53f54a181f958" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:58.659512 25629 raft_consensus.cc:2454] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 0.
I20240502 07:58:58.659487 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6929d2560404acb9bb53f54a181f958" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" is_pre_election: true
I20240502 07:58:58.659634 25497 raft_consensus.cc:2454] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 0.
I20240502 07:58:58.659775 25827 leader_election.cc:302] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:58.659971 26799 raft_consensus.cc:2781] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:58.660032 26799 raft_consensus.cc:482] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:58.660058 26799 raft_consensus.cc:3037] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.660892 26799 raft_consensus.cc:504] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.661197 26799 leader_election.cc:288] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 election: Requested vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:58:58.661495 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6929d2560404acb9bb53f54a181f958" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:58.661545 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6929d2560404acb9bb53f54a181f958" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
I20240502 07:58:58.661664 25629 raft_consensus.cc:3037] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.661680 25497 raft_consensus.cc:3037] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.662386 25497 raft_consensus.cc:2454] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 1.
I20240502 07:58:58.662400 25629 raft_consensus.cc:2454] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 1.
I20240502 07:58:58.662648 25827 leader_election.cc:302] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:58.662792 26799 raft_consensus.cc:2781] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:58.662885 26799 raft_consensus.cc:686] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Becoming Leader. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Running, Role: LEADER
I20240502 07:58:58.663018 26799 consensus_queue.cc:237] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:58.664777 25349 catalog_manager.cc:5484] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9596bfb8d324b4da93a4288771ed177 (127.18.13.132). New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:58.729427 26753 raft_consensus.cc:482] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:58.729578 26753 raft_consensus.cc:504] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.730046 26753 leader_election.cc:288] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:58:58.730530 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1ac05112724e0ea1f84f648598304e" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:58:58.730697 25891 raft_consensus.cc:2454] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfa00c061c748b18772d5f0566944b5 in term 0.
I20240502 07:58:58.731043 25564 leader_election.cc:302] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:58.731240 26753 raft_consensus.cc:2781] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:58.731307 26753 raft_consensus.cc:482] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:58.731334 26753 raft_consensus.cc:3037] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.732482 26753 raft_consensus.cc:504] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.732836 26753 leader_election.cc:288] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 election: Requested vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:58:58.733206 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1ac05112724e0ea1f84f648598304e" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:58:58.733356 25891 raft_consensus.cc:3037] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:58.734113 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1ac05112724e0ea1f84f648598304e" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" is_pre_election: true
I20240502 07:58:58.734148 25760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1ac05112724e0ea1f84f648598304e" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
I20240502 07:58:58.734251 25759 raft_consensus.cc:2454] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfa00c061c748b18772d5f0566944b5 in term 0.
I20240502 07:58:58.756613 25891 raft_consensus.cc:2454] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfa00c061c748b18772d5f0566944b5 in term 1.
I20240502 07:58:58.757103 25564 leader_election.cc:302] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, e9596bfb8d324b4da93a4288771ed177; no voters: 14d8fd1fc3a04d9d967b8033df84f34a
I20240502 07:58:58.757388 26753 raft_consensus.cc:2781] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:58.757606 26753 raft_consensus.cc:686] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Becoming Leader. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Running, Role: LEADER
I20240502 07:58:58.757949 26753 consensus_queue.cc:237] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:58.760375 25349 catalog_manager.cc:5484] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:58.926342 26858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:58.926614 26858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:58.929725 26858 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 07:58:59.027467 25891 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:59.027626 25629 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:59.027984 26548 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:59.028455 26548 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:59.036652 25759 raft_consensus.cc:3037] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.036795 25891 raft_consensus.cc:1261] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:59.037294 26753 consensus_queue.cc:1035] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:59.041647 25759 raft_consensus.cc:1261] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:59.042093 26753 consensus_queue.cc:1035] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:59.047463 25497 raft_consensus.cc:1261] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:59.048733 26799 consensus_queue.cc:1035] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:59.049293 25627 raft_consensus.cc:1261] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:59.049974 26799 consensus_queue.cc:1035] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 132.069 ms
time per row: 0.0660347 ms
I20240502 07:58:59.106731 25347 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40498:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:59.112920 25608 tablet_service.cc:1467] Processing CreateTablet for tablet eb0c33c3e4014eb3b931d64adac60e1a (DEFAULT_TABLE table=rebalance_test_table_6 [id=67841c5c645f4e86970122328fd1238e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.112964 25478 tablet_service.cc:1467] Processing CreateTablet for tablet eb0c33c3e4014eb3b931d64adac60e1a (DEFAULT_TABLE table=rebalance_test_table_6 [id=67841c5c645f4e86970122328fd1238e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.113247 25608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0c33c3e4014eb3b931d64adac60e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.113286 25478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0c33c3e4014eb3b931d64adac60e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.113950 25476 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb2de2942914800bdbe924d45a9b1e7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=67841c5c645f4e86970122328fd1238e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.114008 25740 tablet_service.cc:1467] Processing CreateTablet for tablet eb0c33c3e4014eb3b931d64adac60e1a (DEFAULT_TABLE table=rebalance_test_table_6 [id=67841c5c645f4e86970122328fd1238e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.114185 25476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb2de2942914800bdbe924d45a9b1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.114276 25740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0c33c3e4014eb3b931d64adac60e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.115568 26918 tablet_bootstrap.cc:492] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:59.116011 25871 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb2de2942914800bdbe924d45a9b1e7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=67841c5c645f4e86970122328fd1238e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.116282 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb2de2942914800bdbe924d45a9b1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.116357 26918 tablet_bootstrap.cc:654] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.116667 25738 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb2de2942914800bdbe924d45a9b1e7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=67841c5c645f4e86970122328fd1238e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.116894 25738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb2de2942914800bdbe924d45a9b1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.117246 26919 tablet_bootstrap.cc:492] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:59.117231 25608 tablet_service.cc:1467] Processing CreateTablet for tablet 99c3ddedadbf4c0e8858085b66022342 (DEFAULT_TABLE table=rebalance_test_table_6 [id=67841c5c645f4e86970122328fd1238e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.117446 25608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c3ddedadbf4c0e8858085b66022342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.118089 26919 tablet_bootstrap.cc:654] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.118891 26921 tablet_bootstrap.cc:492] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:59.118958 25739 tablet_service.cc:1467] Processing CreateTablet for tablet 99c3ddedadbf4c0e8858085b66022342 (DEFAULT_TABLE table=rebalance_test_table_6 [id=67841c5c645f4e86970122328fd1238e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.119138 25739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c3ddedadbf4c0e8858085b66022342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.119752 26921 tablet_bootstrap.cc:654] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.120262 26919 tablet_bootstrap.cc:492] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:59.120349 26919 ts_tablet_manager.cc:1367] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:58:59.120796 25477 tablet_service.cc:1467] Processing CreateTablet for tablet 99c3ddedadbf4c0e8858085b66022342 (DEFAULT_TABLE table=rebalance_test_table_6 [id=67841c5c645f4e86970122328fd1238e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.120795 26919 raft_consensus.cc:348] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.120899 26919 raft_consensus.cc:374] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.120930 26919 raft_consensus.cc:729] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.121066 25477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c3ddedadbf4c0e8858085b66022342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.121066 26919 consensus_queue.cc:260] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.121367 26919 ts_tablet_manager.cc:1398] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:59.121577 26919 tablet_bootstrap.cc:492] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:59.121860 26921 tablet_bootstrap.cc:492] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:59.121948 26921 ts_tablet_manager.cc:1367] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:59.122332 26919 tablet_bootstrap.cc:654] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.122319 26921 raft_consensus.cc:348] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.122416 26921 raft_consensus.cc:374] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.122455 26921 raft_consensus.cc:729] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.122594 26921 consensus_queue.cc:260] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.122885 26921 ts_tablet_manager.cc:1398] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:59.123446 26919 tablet_bootstrap.cc:492] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:59.123530 26919 ts_tablet_manager.cc:1367] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:59.123987 26919 raft_consensus.cc:348] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.124105 26919 raft_consensus.cc:374] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.124135 26919 raft_consensus.cc:729] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.124284 26919 consensus_queue.cc:260] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.124692 26919 ts_tablet_manager.cc:1398] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:59.124899 26919 tablet_bootstrap.cc:492] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:59.125520 26918 tablet_bootstrap.cc:492] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:59.125622 26918 ts_tablet_manager.cc:1367] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240502 07:58:59.125775 26919 tablet_bootstrap.cc:654] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.126169 26918 raft_consensus.cc:348] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.126299 26918 raft_consensus.cc:374] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.126332 26918 raft_consensus.cc:729] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.126443 26918 consensus_queue.cc:260] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.127210 26918 ts_tablet_manager.cc:1398] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:59.127390 26918 tablet_bootstrap.cc:492] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:59.128373 26918 tablet_bootstrap.cc:654] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.129482 26918 tablet_bootstrap.cc:492] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:59.129573 26918 ts_tablet_manager.cc:1367] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:59.129622 26919 tablet_bootstrap.cc:492] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:59.129715 26919 ts_tablet_manager.cc:1367] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:58:59.130059 26918 raft_consensus.cc:348] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:59.130125 26919 raft_consensus.cc:348] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:59.130195 26918 raft_consensus.cc:374] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.130221 26919 raft_consensus.cc:374] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.130239 26918 raft_consensus.cc:729] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.130276 26919 raft_consensus.cc:729] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.130373 26918 consensus_queue.cc:260] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:59.130396 26919 consensus_queue.cc:260] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:59.130770 26918 ts_tablet_manager.cc:1398] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:59.131254 26919 ts_tablet_manager.cc:1398] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:59.141831 26927 tablet_bootstrap.cc:492] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:59.142657 26927 tablet_bootstrap.cc:654] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.143620 26927 tablet_bootstrap.cc:492] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:59.143702 26927 ts_tablet_manager.cc:1367] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:59.144179 26927 raft_consensus.cc:348] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.144290 26927 raft_consensus.cc:374] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.144321 26927 raft_consensus.cc:729] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.144456 26927 consensus_queue.cc:260] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.144758 26927 ts_tablet_manager.cc:1398] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:59.144910 26927 tablet_bootstrap.cc:492] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:59.145689 26927 tablet_bootstrap.cc:654] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.146759 26927 tablet_bootstrap.cc:492] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:59.146832 26927 ts_tablet_manager.cc:1367] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:59.147228 26927 raft_consensus.cc:348] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.147344 26927 raft_consensus.cc:374] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.147385 26927 raft_consensus.cc:729] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.147513 26927 consensus_queue.cc:260] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.147853 26927 ts_tablet_manager.cc:1398] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:59.148036 26927 tablet_bootstrap.cc:492] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:59.148811 26927 tablet_bootstrap.cc:654] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.149966 26927 tablet_bootstrap.cc:492] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:59.150036 26927 ts_tablet_manager.cc:1367] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:59.150359 26927 raft_consensus.cc:348] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:59.150447 26927 raft_consensus.cc:374] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.150475 26927 raft_consensus.cc:729] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.150635 26927 consensus_queue.cc:260] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:59.150915 26927 ts_tablet_manager.cc:1398] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:59.291469 26837 raft_consensus.cc:482] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:59.291638 26837 raft_consensus.cc:504] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.292122 26837 leader_election.cc:288] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:59.292577 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb2de2942914800bdbe924d45a9b1e7" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" is_pre_election: true
I20240502 07:58:59.292577 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb2de2942914800bdbe924d45a9b1e7" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:58:59.292716 25759 raft_consensus.cc:2454] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:59.292726 25891 raft_consensus.cc:2454] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:59.293049 25433 leader_election.cc:302] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3be11f0fe304e0f8e008dc1e4c41f70, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:59.293217 26837 raft_consensus.cc:2781] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:59.293293 26837 raft_consensus.cc:482] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:59.293329 26837 raft_consensus.cc:3037] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.294133 26837 raft_consensus.cc:504] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.294481 26837 leader_election.cc:288] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Requested vote from peers 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:59.294840 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb2de2942914800bdbe924d45a9b1e7" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:58:59.294857 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb2de2942914800bdbe924d45a9b1e7" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
I20240502 07:58:59.294951 25759 raft_consensus.cc:3037] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.294955 25891 raft_consensus.cc:3037] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.295691 25759 raft_consensus.cc:2454] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:59.295713 25891 raft_consensus.cc:2454] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:59.295998 25432 leader_election.cc:302] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:59.296159 26837 raft_consensus.cc:2781] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:59.296378 26837 raft_consensus.cc:686] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Becoming Leader. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Running, Role: LEADER
I20240502 07:58:59.296523 26837 consensus_queue.cc:237] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.298312 25347 catalog_manager.cc:5484] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:59.304296 26898 raft_consensus.cc:482] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:59.304419 26898 raft_consensus.cc:504] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:59.304836 26898 leader_election.cc:288] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:59.305220 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c3ddedadbf4c0e8858085b66022342" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" is_pre_election: true
I20240502 07:58:59.305307 25627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c3ddedadbf4c0e8858085b66022342" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:59.305389 25497 raft_consensus.cc:2454] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:59.305446 25627 raft_consensus.cc:2454] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:59.305689 25696 leader_election.cc:302] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:59.305872 26898 raft_consensus.cc:2781] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:59.305948 26898 raft_consensus.cc:482] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:59.305996 26898 raft_consensus.cc:3037] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.306869 26898 raft_consensus.cc:504] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:59.307230 26898 leader_election.cc:288] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Requested vote from peers c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:58:59.307587 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c3ddedadbf4c0e8858085b66022342" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
I20240502 07:58:59.307655 25627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c3ddedadbf4c0e8858085b66022342" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:59.307716 25497 raft_consensus.cc:3037] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.307821 25627 raft_consensus.cc:3037] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.308600 25497 raft_consensus.cc:2454] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:58:59.308801 25627 raft_consensus.cc:2454] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:58:59.308918 25696 leader_election.cc:302] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:59.309099 26898 raft_consensus.cc:2781] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:59.309186 26898 raft_consensus.cc:686] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Becoming Leader. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Running, Role: LEADER
I20240502 07:58:59.309350 26898 consensus_queue.cc:237] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:58:59.311187 25347 catalog_manager.cc:5484] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: term changed from 0 to 1, leader changed from <none> to 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131). New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:59.345305 26837 raft_consensus.cc:482] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:59.345472 26837 raft_consensus.cc:504] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.345911 26837 leader_election.cc:288] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:58:59.346426 25627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0c33c3e4014eb3b931d64adac60e1a" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:59.346428 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0c33c3e4014eb3b931d64adac60e1a" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" is_pre_election: true
I20240502 07:58:59.346571 25627 raft_consensus.cc:2454] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:59.346575 25759 raft_consensus.cc:2454] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:59.346896 25432 leader_election.cc:302] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:59.347059 26837 raft_consensus.cc:2781] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:59.347113 26837 raft_consensus.cc:482] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:59.347146 26837 raft_consensus.cc:3037] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.347971 26837 raft_consensus.cc:504] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.348307 26837 leader_election.cc:288] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Requested vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:58:59.348621 25627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0c33c3e4014eb3b931d64adac60e1a" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:59.348645 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0c33c3e4014eb3b931d64adac60e1a" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
I20240502 07:58:59.348722 25627 raft_consensus.cc:3037] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.348753 25759 raft_consensus.cc:3037] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.349450 25627 raft_consensus.cc:2454] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:59.349504 25759 raft_consensus.cc:2454] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:59.349736 25434 leader_election.cc:302] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:59.349898 26837 raft_consensus.cc:2781] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:59.349989 26837 raft_consensus.cc:686] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Becoming Leader. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Running, Role: LEADER
I20240502 07:58:59.350116 26837 consensus_queue.cc:237] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.352113 25347 catalog_manager.cc:5484] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:59.474244 26935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:59.474534 26935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:59.477816 26935 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 07:58:59.571027 25759 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:59.571089 25627 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:59.571556 26931 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:59.572119 26837 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:59.573004 25891 raft_consensus.cc:1261] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:59.573112 25497 raft_consensus.cc:1261] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:59.573523 26728 consensus_queue.cc:1035] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:59.574003 25627 raft_consensus.cc:1261] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:59.574127 26931 consensus_queue.cc:1035] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:59.574815 26898 consensus_queue.cc:1035] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:59.582928 26892 mvcc.cc:205] Tried to move back new op lower bound from 7023152085275922432 to 7023152084379181056. Current Snapshot: MvccSnapshot[applied={T|T < 7023152085275922432}]
I20240502 07:58:59.588805 25759 raft_consensus.cc:1261] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:59.601356 26966 mvcc.cc:205] Tried to move back new op lower bound from 7023152085285761024 to 7023152084212559872. Current Snapshot: MvccSnapshot[applied={T|T < 7023152085285761024}]
I20240502 07:58:59.601660 26931 consensus_queue.cc:1035] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:59.604614 26982 mvcc.cc:205] Tried to move back new op lower bound from 7023152085275922432 to 7023152084379181056. Current Snapshot: MvccSnapshot[applied={T|T < 7023152085275922432}]
INSERT report
rows total: 2000
time total: 111.239 ms
time per row: 0.0556197 ms
I20240502 07:58:59.646373 25349 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40498:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:59.651983 25608 tablet_service.cc:1467] Processing CreateTablet for tablet dbfd23aa383f413cbe4a5ccd8146d2b6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=89a9dd84e8b5432a81ef7767804381d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.652336 25608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfd23aa383f413cbe4a5ccd8146d2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.652876 25871 tablet_service.cc:1467] Processing CreateTablet for tablet dbfd23aa383f413cbe4a5ccd8146d2b6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=89a9dd84e8b5432a81ef7767804381d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.653151 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfd23aa383f413cbe4a5ccd8146d2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.653268 25609 tablet_service.cc:1467] Processing CreateTablet for tablet 273e203d4ed44c43859f08e9bec08a9d (DEFAULT_TABLE table=rebalance_test_table_7 [id=89a9dd84e8b5432a81ef7767804381d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.653333 25478 tablet_service.cc:1467] Processing CreateTablet for tablet dbfd23aa383f413cbe4a5ccd8146d2b6 (DEFAULT_TABLE table=rebalance_test_table_7 [id=89a9dd84e8b5432a81ef7767804381d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.653530 25609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273e203d4ed44c43859f08e9bec08a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.653563 25478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfd23aa383f413cbe4a5ccd8146d2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.654969 26987 tablet_bootstrap.cc:492] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:59.655817 25477 tablet_service.cc:1467] Processing CreateTablet for tablet d8e50f3d37d842868900a4cff5bf3601 (DEFAULT_TABLE table=rebalance_test_table_7 [id=89a9dd84e8b5432a81ef7767804381d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.655962 26987 tablet_bootstrap.cc:654] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.656069 25477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e50f3d37d842868900a4cff5bf3601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.656137 25739 tablet_service.cc:1467] Processing CreateTablet for tablet d8e50f3d37d842868900a4cff5bf3601 (DEFAULT_TABLE table=rebalance_test_table_7 [id=89a9dd84e8b5432a81ef7767804381d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.656365 26988 tablet_bootstrap.cc:492] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:59.656436 25739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e50f3d37d842868900a4cff5bf3601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.656975 26987 tablet_bootstrap.cc:492] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:59.657063 26987 ts_tablet_manager.cc:1367] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:59.657472 26987 raft_consensus.cc:348] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:59.657600 26987 raft_consensus.cc:374] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.657644 26987 raft_consensus.cc:729] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.657786 26987 consensus_queue.cc:260] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:59.657995 26988 tablet_bootstrap.cc:654] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.658072 26987 ts_tablet_manager.cc:1398] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:59.658813 25738 tablet_service.cc:1467] Processing CreateTablet for tablet 273e203d4ed44c43859f08e9bec08a9d (DEFAULT_TABLE table=rebalance_test_table_7 [id=89a9dd84e8b5432a81ef7767804381d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.659054 25738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273e203d4ed44c43859f08e9bec08a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.659157 26988 tablet_bootstrap.cc:492] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:59.659235 26988 ts_tablet_manager.cc:1367] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:59.659389 26991 tablet_bootstrap.cc:492] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:59.659662 26988 raft_consensus.cc:348] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.660041 25870 tablet_service.cc:1467] Processing CreateTablet for tablet 273e203d4ed44c43859f08e9bec08a9d (DEFAULT_TABLE table=rebalance_test_table_7 [id=89a9dd84e8b5432a81ef7767804381d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.660274 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273e203d4ed44c43859f08e9bec08a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.659765 26988 raft_consensus.cc:374] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.660516 26988 raft_consensus.cc:729] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.660650 26988 consensus_queue.cc:260] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.660933 26988 ts_tablet_manager.cc:1398] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:59.661274 26991 tablet_bootstrap.cc:654] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.662631 26988 tablet_bootstrap.cc:492] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:58:59.662962 26989 tablet_bootstrap.cc:492] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:59.663175 25871 tablet_service.cc:1467] Processing CreateTablet for tablet d8e50f3d37d842868900a4cff5bf3601 (DEFAULT_TABLE table=rebalance_test_table_7 [id=89a9dd84e8b5432a81ef7767804381d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:59.663300 26987 tablet_bootstrap.cc:492] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:59.663408 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e50f3d37d842868900a4cff5bf3601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:59.663515 26988 tablet_bootstrap.cc:654] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.663954 26989 tablet_bootstrap.cc:654] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.664201 26987 tablet_bootstrap.cc:654] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.665032 26988 tablet_bootstrap.cc:492] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:58:59.665113 26988 ts_tablet_manager.cc:1367] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:59.665203 26989 tablet_bootstrap.cc:492] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:59.665293 26989 ts_tablet_manager.cc:1367] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:59.665365 26987 tablet_bootstrap.cc:492] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:59.665434 26987 ts_tablet_manager.cc:1367] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:59.665570 26988 raft_consensus.cc:348] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:59.665684 26988 raft_consensus.cc:374] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.665716 26988 raft_consensus.cc:729] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.665756 26991 tablet_bootstrap.cc:492] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:59.665835 26991 ts_tablet_manager.cc:1367] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 07:58:59.665833 26988 consensus_queue.cc:260] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:59.665834 26987 raft_consensus.cc:348] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.665915 26987 raft_consensus.cc:374] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.665948 26987 raft_consensus.cc:729] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.665961 26989 raft_consensus.cc:348] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:59.666059 26989 raft_consensus.cc:374] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.666088 26989 raft_consensus.cc:729] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.666050 26987 consensus_queue.cc:260] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.666107 26988 ts_tablet_manager.cc:1398] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:59.666263 26989 consensus_queue.cc:260] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:59.666280 26991 raft_consensus.cc:348] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.666345 26987 ts_tablet_manager.cc:1398] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:59.666369 26991 raft_consensus.cc:374] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.666492 26991 raft_consensus.cc:729] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.666518 26987 tablet_bootstrap.cc:492] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:58:59.666577 26989 ts_tablet_manager.cc:1398] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:59.666718 26989 tablet_bootstrap.cc:492] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:58:59.666666 26991 consensus_queue.cc:260] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.667019 26991 ts_tablet_manager.cc:1398] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:59.667177 26991 tablet_bootstrap.cc:492] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:58:59.667323 26987 tablet_bootstrap.cc:654] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.668128 26991 tablet_bootstrap.cc:654] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.669523 26989 tablet_bootstrap.cc:654] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:59.670686 26987 tablet_bootstrap.cc:492] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:58:59.670698 26989 tablet_bootstrap.cc:492] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:58:59.670786 26989 ts_tablet_manager.cc:1367] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:59.670791 26987 ts_tablet_manager.cc:1367] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 07:58:59.671180 26987 raft_consensus.cc:348] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.671272 26987 raft_consensus.cc:374] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.671303 26987 raft_consensus.cc:729] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.671273 26989 raft_consensus.cc:348] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.671403 26989 raft_consensus.cc:374] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.671442 26989 raft_consensus.cc:729] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.671429 26987 consensus_queue.cc:260] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.671562 26989 consensus_queue.cc:260] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.671722 26987 ts_tablet_manager.cc:1398] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:59.671970 26989 ts_tablet_manager.cc:1398] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:59.672492 26991 tablet_bootstrap.cc:492] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:58:59.672596 26991 ts_tablet_manager.cc:1367] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 07:58:59.673272 26991 raft_consensus.cc:348] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.673403 26991 raft_consensus.cc:374] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:59.673449 26991 raft_consensus.cc:729] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:58:59.673581 26991 consensus_queue.cc:260] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.673884 26991 ts_tablet_manager.cc:1398] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:59.737859 26712 raft_consensus.cc:482] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:59.738031 26712 raft_consensus.cc:504] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:59.738517 26712 leader_election.cc:288] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:58:59.739106 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfd23aa383f413cbe4a5ccd8146d2b6" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:59.739106 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfd23aa383f413cbe4a5ccd8146d2b6" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" is_pre_election: true
I20240502 07:58:59.739275 25497 raft_consensus.cc:2454] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 0.
I20240502 07:58:59.739275 25629 raft_consensus.cc:2454] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 0.
I20240502 07:58:59.739593 25827 leader_election.cc:302] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:59.739777 26712 raft_consensus.cc:2781] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:59.739881 26712 raft_consensus.cc:482] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:59.739929 26712 raft_consensus.cc:3037] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.741070 26712 raft_consensus.cc:504] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:59.741400 26712 leader_election.cc:288] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 election: Requested vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:58:59.741781 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfd23aa383f413cbe4a5ccd8146d2b6" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:59.741909 25629 raft_consensus.cc:3037] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.741873 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbfd23aa383f413cbe4a5ccd8146d2b6" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
I20240502 07:58:59.742007 25497 raft_consensus.cc:3037] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.742880 25629 raft_consensus.cc:2454] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 1.
I20240502 07:58:59.742988 25497 raft_consensus.cc:2454] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 1.
I20240502 07:58:59.743166 25827 leader_election.cc:302] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:59.743340 26712 raft_consensus.cc:2781] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:59.743521 26712 raft_consensus.cc:686] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Becoming Leader. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Running, Role: LEADER
I20240502 07:58:59.743666 26712 consensus_queue.cc:237] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:58:59.745360 25353 catalog_manager.cc:5484] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9596bfb8d324b4da93a4288771ed177 (127.18.13.132). New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:59.820120 26931 raft_consensus.cc:482] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:59.820279 26931 raft_consensus.cc:504] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.820748 26931 leader_election.cc:288] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:58:59.821200 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e50f3d37d842868900a4cff5bf3601" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:58:59.821246 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e50f3d37d842868900a4cff5bf3601" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" is_pre_election: true
I20240502 07:58:59.821350 25891 raft_consensus.cc:2454] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:59.821384 25759 raft_consensus.cc:2454] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 0.
I20240502 07:58:59.821671 25433 leader_election.cc:302] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3be11f0fe304e0f8e008dc1e4c41f70, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:59.821945 26931 raft_consensus.cc:2781] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:59.822031 26931 raft_consensus.cc:482] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:59.822072 26931 raft_consensus.cc:3037] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.823204 26931 raft_consensus.cc:504] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.823624 26931 leader_election.cc:288] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Requested vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:58:59.824048 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e50f3d37d842868900a4cff5bf3601" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:58:59.824048 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8e50f3d37d842868900a4cff5bf3601" candidate_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
I20240502 07:58:59.824191 25891 raft_consensus.cc:3037] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.824191 25759 raft_consensus.cc:3037] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.825220 25891 raft_consensus.cc:2454] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:59.825220 25759 raft_consensus.cc:2454] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3be11f0fe304e0f8e008dc1e4c41f70 in term 1.
I20240502 07:58:59.825585 25432 leader_election.cc:302] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:58:59.825795 26931 raft_consensus.cc:2781] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:59.825881 26931 raft_consensus.cc:686] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Becoming Leader. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Running, Role: LEADER
I20240502 07:58:59.826030 26931 consensus_queue.cc:237] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:58:59.828135 25353 catalog_manager.cc:5484] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:59.892290 26728 raft_consensus.cc:482] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:59.892437 26728 raft_consensus.cc:504] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.892861 26728 leader_election.cc:288] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:59.893349 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273e203d4ed44c43859f08e9bec08a9d" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:58:59.893405 25629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273e203d4ed44c43859f08e9bec08a9d" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:58:59.893518 25891 raft_consensus.cc:2454] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:59.893590 25629 raft_consensus.cc:2454] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:58:59.893862 25695 leader_election.cc:302] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:58:59.894168 26728 raft_consensus.cc:2781] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:59.894258 26728 raft_consensus.cc:482] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:59.894307 26728 raft_consensus.cc:3037] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.895161 26728 raft_consensus.cc:504] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.895524 26728 leader_election.cc:288] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Requested vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:58:59.895902 25627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273e203d4ed44c43859f08e9bec08a9d" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:58:59.895927 25891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273e203d4ed44c43859f08e9bec08a9d" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:58:59.896018 25627 raft_consensus.cc:3037] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.896047 25891 raft_consensus.cc:3037] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:59.896796 25627 raft_consensus.cc:2454] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:58:59.896878 25891 raft_consensus.cc:2454] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:58:59.897109 25696 leader_election.cc:302] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, 14d8fd1fc3a04d9d967b8033df84f34a; no voters:
I20240502 07:58:59.897241 26728 raft_consensus.cc:2781] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:59.897341 26728 raft_consensus.cc:686] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Becoming Leader. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Running, Role: LEADER
I20240502 07:58:59.897485 26728 consensus_queue.cc:237] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:58:59.899088 25353 catalog_manager.cc:5484] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: term changed from 0 to 1, leader changed from <none> to 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131). New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:00.075363 27004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:00.075699 27004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:00.079313 27004 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 07:59:00.178365 25759 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:00.178802 26931 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:00.179589 25891 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:00.180502 25497 raft_consensus.cc:1261] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:00.180747 25627 raft_consensus.cc:1261] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:00.181111 26931 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:00.181138 26712 consensus_queue.cc:1035] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:00.181265 27000 consensus_queue.cc:1035] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:00.198076 25890 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:00.198449 25627 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:00.198614 26728 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:00.200016 26728 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:00.232645 27043 mvcc.cc:205] Tried to move back new op lower bound from 7023152087847047168 to 7023152086621237248. Current Snapshot: MvccSnapshot[applied={T|T < 7023152087847047168}]
INSERT report
rows total: 2000
time total: 139.096 ms
time per row: 0.0695482 ms
I20240502 07:59:00.265266 25354 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40498:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:00.270952 25871 tablet_service.cc:1467] Processing CreateTablet for tablet 037fed57490949bbbafa11ebb44d4bb8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8aaedf9499044a93ab9419f10e2342e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.270994 25477 tablet_service.cc:1467] Processing CreateTablet for tablet 037fed57490949bbbafa11ebb44d4bb8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8aaedf9499044a93ab9419f10e2342e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.271286 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037fed57490949bbbafa11ebb44d4bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.271296 25477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037fed57490949bbbafa11ebb44d4bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.273685 27065 tablet_bootstrap.cc:492] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:59:00.273839 25739 tablet_service.cc:1467] Processing CreateTablet for tablet f21d6014b91c45df96325017b57f19e3 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8aaedf9499044a93ab9419f10e2342e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.274187 25739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21d6014b91c45df96325017b57f19e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.274298 27064 tablet_bootstrap.cc:492] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:59:00.274776 25478 tablet_service.cc:1467] Processing CreateTablet for tablet f21d6014b91c45df96325017b57f19e3 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8aaedf9499044a93ab9419f10e2342e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.275024 25478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21d6014b91c45df96325017b57f19e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.275355 27064 tablet_bootstrap.cc:654] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.276639 27065 tablet_bootstrap.cc:654] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.273839 25738 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad22d2e464d4defab6bd0061e6058f8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8aaedf9499044a93ab9419f10e2342e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.277074 25738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad22d2e464d4defab6bd0061e6058f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.277655 25609 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad22d2e464d4defab6bd0061e6058f8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8aaedf9499044a93ab9419f10e2342e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.277962 25609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad22d2e464d4defab6bd0061e6058f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.278533 25477 tablet_service.cc:1467] Processing CreateTablet for tablet 1ad22d2e464d4defab6bd0061e6058f8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8aaedf9499044a93ab9419f10e2342e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.278600 27064 tablet_bootstrap.cc:492] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:59:00.278743 27064 ts_tablet_manager.cc:1367] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 07:59:00.278808 25477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad22d2e464d4defab6bd0061e6058f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.279567 27064 raft_consensus.cc:348] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.279587 25871 tablet_service.cc:1467] Processing CreateTablet for tablet f21d6014b91c45df96325017b57f19e3 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8aaedf9499044a93ab9419f10e2342e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.280016 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21d6014b91c45df96325017b57f19e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.277655 25608 tablet_service.cc:1467] Processing CreateTablet for tablet 037fed57490949bbbafa11ebb44d4bb8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8aaedf9499044a93ab9419f10e2342e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.280324 27069 tablet_bootstrap.cc:492] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:59:00.280400 25608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037fed57490949bbbafa11ebb44d4bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.281415 27069 tablet_bootstrap.cc:654] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.281474 27067 tablet_bootstrap.cc:492] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:59:00.282438 27067 tablet_bootstrap.cc:654] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.282522 27064 raft_consensus.cc:374] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.282631 27064 raft_consensus.cc:729] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.282764 27069 tablet_bootstrap.cc:492] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:59:00.282856 27069 ts_tablet_manager.cc:1367] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:00.283032 27065 tablet_bootstrap.cc:492] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:59:00.283118 27065 ts_tablet_manager.cc:1367] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 07:59:00.283314 27069 raft_consensus.cc:348] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.283433 27069 raft_consensus.cc:374] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.283476 27069 raft_consensus.cc:729] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.283558 27065 raft_consensus.cc:348] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.283601 27069 consensus_queue.cc:260] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.283680 27065 raft_consensus.cc:374] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.283717 27065 raft_consensus.cc:729] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.283885 27065 consensus_queue.cc:260] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.283954 27069 ts_tablet_manager.cc:1398] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:00.283313 27064 consensus_queue.cc:260] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.284116 27069 tablet_bootstrap.cc:492] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:59:00.285001 27069 tablet_bootstrap.cc:654] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.285653 27064 ts_tablet_manager.cc:1398] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240502 07:59:00.285779 27065 ts_tablet_manager.cc:1398] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:59:00.285944 27064 tablet_bootstrap.cc:492] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:59:00.285967 27065 tablet_bootstrap.cc:492] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:59:00.286231 27067 tablet_bootstrap.cc:492] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:59:00.286337 27067 ts_tablet_manager.cc:1367] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:59:00.286835 27067 raft_consensus.cc:348] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.286939 27065 tablet_bootstrap.cc:654] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.286972 27067 raft_consensus.cc:374] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.287014 27067 raft_consensus.cc:729] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.287153 27067 consensus_queue.cc:260] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.287431 27064 tablet_bootstrap.cc:654] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.288182 27067 ts_tablet_manager.cc:1398] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:00.288367 27067 tablet_bootstrap.cc:492] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:59:00.289300 27067 tablet_bootstrap.cc:654] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.289507 27065 tablet_bootstrap.cc:492] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:59:00.289645 27065 ts_tablet_manager.cc:1367] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:00.290166 27065 raft_consensus.cc:348] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.290306 27065 raft_consensus.cc:374] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.290346 27065 raft_consensus.cc:729] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.290479 27065 consensus_queue.cc:260] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.290783 27065 ts_tablet_manager.cc:1398] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:00.290948 27065 tablet_bootstrap.cc:492] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:59:00.291633 27065 tablet_bootstrap.cc:654] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.292820 27065 tablet_bootstrap.cc:492] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:59:00.292894 27069 tablet_bootstrap.cc:492] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:59:00.292920 27065 ts_tablet_manager.cc:1367] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:00.292989 27069 ts_tablet_manager.cc:1367] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 07:59:00.293465 27065 raft_consensus.cc:348] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.293550 27065 raft_consensus.cc:374] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.293571 27065 raft_consensus.cc:729] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.293473 27069 raft_consensus.cc:348] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.293605 27069 raft_consensus.cc:374] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.293637 27069 raft_consensus.cc:729] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.293673 27065 consensus_queue.cc:260] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.293784 27069 consensus_queue.cc:260] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.294078 27069 ts_tablet_manager.cc:1398] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:00.294934 27064 tablet_bootstrap.cc:492] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:59:00.295029 27064 ts_tablet_manager.cc:1367] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240502 07:59:00.295130 27067 tablet_bootstrap.cc:492] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:59:00.295217 27067 ts_tablet_manager.cc:1367] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 07:59:00.295528 27064 raft_consensus.cc:348] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.295627 27064 raft_consensus.cc:374] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.295655 27064 raft_consensus.cc:729] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.295727 27067 raft_consensus.cc:348] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.295759 27064 consensus_queue.cc:260] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.295867 27067 raft_consensus.cc:374] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.295900 27067 raft_consensus.cc:729] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.296010 27067 consensus_queue.cc:260] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.296087 27064 ts_tablet_manager.cc:1398] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:00.294472 27065 ts_tablet_manager.cc:1398] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:00.296272 27067 ts_tablet_manager.cc:1398] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:00.347767 26898 raft_consensus.cc:482] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:00.347936 26898 raft_consensus.cc:504] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.348381 26898 leader_election.cc:288] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:00.348847 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21d6014b91c45df96325017b57f19e3" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:59:00.348850 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21d6014b91c45df96325017b57f19e3" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" is_pre_election: true
I20240502 07:59:00.348977 25890 raft_consensus.cc:2454] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:59:00.348999 25497 raft_consensus.cc:2454] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 0.
I20240502 07:59:00.349331 25695 leader_election.cc:302] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:59:00.349498 26898 raft_consensus.cc:2781] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:00.349562 26898 raft_consensus.cc:482] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:00.349591 26898 raft_consensus.cc:3037] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.350533 26898 raft_consensus.cc:504] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.350898 26898 leader_election.cc:288] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Requested vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:00.351271 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21d6014b91c45df96325017b57f19e3" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:59:00.351401 25890 raft_consensus.cc:3037] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.351516 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f21d6014b91c45df96325017b57f19e3" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
I20240502 07:59:00.351651 25497 raft_consensus.cc:3037] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.352345 25890 raft_consensus.cc:2454] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:59:00.352473 25497 raft_consensus.cc:2454] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 1.
I20240502 07:59:00.352669 25695 leader_election.cc:302] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:59:00.352828 26898 raft_consensus.cc:2781] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:00.352929 26898 raft_consensus.cc:686] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Becoming Leader. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Running, Role: LEADER
I20240502 07:59:00.353076 26898 consensus_queue.cc:237] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.354820 25352 catalog_manager.cc:5484] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: term changed from 0 to 1, leader changed from <none> to 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131). New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:00.364212 26915 raft_consensus.cc:482] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:00.364341 26915 raft_consensus.cc:504] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.364739 26915 leader_election.cc:288] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205), c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:00.365119 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad22d2e464d4defab6bd0061e6058f8" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" is_pre_election: true
I20240502 07:59:00.365130 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad22d2e464d4defab6bd0061e6058f8" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" is_pre_election: true
I20240502 07:59:00.365264 25497 raft_consensus.cc:2454] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfa00c061c748b18772d5f0566944b5 in term 0.
I20240502 07:59:00.365263 25759 raft_consensus.cc:2454] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfa00c061c748b18772d5f0566944b5 in term 0.
I20240502 07:59:00.365548 25565 leader_election.cc:302] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:59:00.365712 26915 raft_consensus.cc:2781] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:00.365769 26915 raft_consensus.cc:482] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:00.365797 26915 raft_consensus.cc:3037] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.366684 26915 raft_consensus.cc:504] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.367043 26915 leader_election.cc:288] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 election: Requested vote from peers 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205), c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:00.367369 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad22d2e464d4defab6bd0061e6058f8" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
I20240502 07:59:00.367403 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ad22d2e464d4defab6bd0061e6058f8" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
I20240502 07:59:00.367493 25759 raft_consensus.cc:3037] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.367507 25497 raft_consensus.cc:3037] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.368466 25759 raft_consensus.cc:2454] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfa00c061c748b18772d5f0566944b5 in term 1.
I20240502 07:59:00.368487 25497 raft_consensus.cc:2454] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfa00c061c748b18772d5f0566944b5 in term 1.
I20240502 07:59:00.368750 25565 leader_election.cc:302] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:59:00.368942 26915 raft_consensus.cc:2781] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:00.369148 26915 raft_consensus.cc:686] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Becoming Leader. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Running, Role: LEADER
I20240502 07:59:00.369297 26915 consensus_queue.cc:237] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.370998 25352 catalog_manager.cc:5484] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:00.384640 27058 raft_consensus.cc:482] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:00.384799 27058 raft_consensus.cc:504] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.385195 27058 leader_election.cc:288] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:00.385635 25627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037fed57490949bbbafa11ebb44d4bb8" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:59:00.385668 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037fed57490949bbbafa11ebb44d4bb8" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" is_pre_election: true
I20240502 07:59:00.385802 25497 raft_consensus.cc:2454] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 0.
I20240502 07:59:00.385802 25627 raft_consensus.cc:2454] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 0.
I20240502 07:59:00.386073 25827 leader_election.cc:302] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3be11f0fe304e0f8e008dc1e4c41f70, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:59:00.386289 27058 raft_consensus.cc:2781] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:00.386365 27058 raft_consensus.cc:482] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:00.386405 27058 raft_consensus.cc:3037] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.387409 27058 raft_consensus.cc:504] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.387703 27058 leader_election.cc:288] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 election: Requested vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:00.388150 25627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037fed57490949bbbafa11ebb44d4bb8" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:59:00.388165 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037fed57490949bbbafa11ebb44d4bb8" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
I20240502 07:59:00.388283 25497 raft_consensus.cc:3037] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.388298 25627 raft_consensus.cc:3037] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.389222 25627 raft_consensus.cc:2454] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 1.
I20240502 07:59:00.389252 25497 raft_consensus.cc:2454] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 1.
I20240502 07:59:00.389528 25827 leader_election.cc:302] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3be11f0fe304e0f8e008dc1e4c41f70, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:59:00.389668 27058 raft_consensus.cc:2781] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:00.389751 27058 raft_consensus.cc:686] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Becoming Leader. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Running, Role: LEADER
I20240502 07:59:00.389914 27058 consensus_queue.cc:237] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } }
I20240502 07:59:00.391860 25352 catalog_manager.cc:5484] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9596bfb8d324b4da93a4288771ed177 (127.18.13.132). New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:00.525060 27081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:00.525308 27081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:00.528329 27081 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 07:59:00.606985 25627 raft_consensus.cc:1261] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:00.607143 25890 raft_consensus.cc:1261] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:00.608139 25498 raft_consensus.cc:1261] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:00.608191 25497 raft_consensus.cc:1261] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:00.608551 26898 consensus_queue.cc:1035] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:00.608686 26728 consensus_queue.cc:1035] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:00.609074 27000 consensus_queue.cc:1035] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:00.615065 25498 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:00.615759 25759 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:00.616132 26915 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:00.616344 27078 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:00.626426 27000 consensus_queue.cc:1035] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.325 ms
time per row: 0.0621625 ms
I20240502 07:59:00.693552 25352 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40498:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:00.698850 25608 tablet_service.cc:1467] Processing CreateTablet for tablet c81a7795712e45b5a7fd97450a9f06a7 (DEFAULT_TABLE table=rebalance_test_table_9 [id=10514d1b971c4d5c9e2b1f433f40d2c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.699025 25738 tablet_service.cc:1467] Processing CreateTablet for tablet c81a7795712e45b5a7fd97450a9f06a7 (DEFAULT_TABLE table=rebalance_test_table_9 [id=10514d1b971c4d5c9e2b1f433f40d2c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.699189 25608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c81a7795712e45b5a7fd97450a9f06a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.699311 25738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c81a7795712e45b5a7fd97450a9f06a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.699728 25609 tablet_service.cc:1467] Processing CreateTablet for tablet 843d1c1d6384476983749a78f65a5c74 (DEFAULT_TABLE table=rebalance_test_table_9 [id=10514d1b971c4d5c9e2b1f433f40d2c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.699821 25477 tablet_service.cc:1467] Processing CreateTablet for tablet 21a8d5798c8943658113d67c88f4ad25 (DEFAULT_TABLE table=rebalance_test_table_9 [id=10514d1b971c4d5c9e2b1f433f40d2c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.699963 25609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843d1c1d6384476983749a78f65a5c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.700071 25477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a8d5798c8943658113d67c88f4ad25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.700076 25607 tablet_service.cc:1467] Processing CreateTablet for tablet 21a8d5798c8943658113d67c88f4ad25 (DEFAULT_TABLE table=rebalance_test_table_9 [id=10514d1b971c4d5c9e2b1f433f40d2c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.700299 25607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a8d5798c8943658113d67c88f4ad25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.702049 27069 tablet_bootstrap.cc:492] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:59:00.702134 27065 tablet_bootstrap.cc:492] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70: Bootstrap starting.
I20240502 07:59:00.702401 25871 tablet_service.cc:1467] Processing CreateTablet for tablet c81a7795712e45b5a7fd97450a9f06a7 (DEFAULT_TABLE table=rebalance_test_table_9 [id=10514d1b971c4d5c9e2b1f433f40d2c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.702661 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c81a7795712e45b5a7fd97450a9f06a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.702847 27069 tablet_bootstrap.cc:654] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.702982 27065 tablet_bootstrap.cc:654] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.703547 25870 tablet_service.cc:1467] Processing CreateTablet for tablet 843d1c1d6384476983749a78f65a5c74 (DEFAULT_TABLE table=rebalance_test_table_9 [id=10514d1b971c4d5c9e2b1f433f40d2c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.703768 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843d1c1d6384476983749a78f65a5c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.704596 27064 tablet_bootstrap.cc:492] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:59:00.705181 25739 tablet_service.cc:1467] Processing CreateTablet for tablet 843d1c1d6384476983749a78f65a5c74 (DEFAULT_TABLE table=rebalance_test_table_9 [id=10514d1b971c4d5c9e2b1f433f40d2c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.705489 27064 tablet_bootstrap.cc:654] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.705514 25739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843d1c1d6384476983749a78f65a5c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.706385 27065 tablet_bootstrap.cc:492] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70: No bootstrap required, opened a new log
I20240502 07:59:00.706490 27065 ts_tablet_manager.cc:1367] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:00.706838 27065 raft_consensus.cc:348] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:59:00.706983 27065 raft_consensus.cc:374] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.707028 27065 raft_consensus.cc:729] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.707141 27065 consensus_queue.cc:260] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:59:00.707286 27067 tablet_bootstrap.cc:492] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:59:00.707417 27065 ts_tablet_manager.cc:1398] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:00.707911 27069 tablet_bootstrap.cc:492] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:59:00.707994 27069 ts_tablet_manager.cc:1367] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:59:00.708287 27067 tablet_bootstrap.cc:654] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.708410 27069 raft_consensus.cc:348] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:59:00.708458 25870 tablet_service.cc:1467] Processing CreateTablet for tablet 21a8d5798c8943658113d67c88f4ad25 (DEFAULT_TABLE table=rebalance_test_table_9 [id=10514d1b971c4d5c9e2b1f433f40d2c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:00.708513 27069 raft_consensus.cc:374] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.708544 27069 raft_consensus.cc:729] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.708675 25870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a8d5798c8943658113d67c88f4ad25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:00.708655 27069 consensus_queue.cc:260] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:59:00.708787 27064 tablet_bootstrap.cc:492] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:59:00.708868 27064 ts_tablet_manager.cc:1367] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 07:59:00.708935 27069 ts_tablet_manager.cc:1398] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:00.709095 27069 tablet_bootstrap.cc:492] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:59:00.709254 27064 raft_consensus.cc:348] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:59:00.709347 27064 raft_consensus.cc:374] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.709374 27064 raft_consensus.cc:729] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.709481 27064 consensus_queue.cc:260] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:59:00.709731 27064 ts_tablet_manager.cc:1398] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:00.709898 27064 tablet_bootstrap.cc:492] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:59:00.709933 27069 tablet_bootstrap.cc:654] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.710518 27067 tablet_bootstrap.cc:492] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:59:00.710592 27067 ts_tablet_manager.cc:1367] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:00.711006 27067 raft_consensus.cc:348] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:59:00.711110 27067 raft_consensus.cc:374] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.711159 27067 raft_consensus.cc:729] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.711333 27069 tablet_bootstrap.cc:492] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:59:00.711347 27064 tablet_bootstrap.cc:654] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.711287 27067 consensus_queue.cc:260] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:59:00.711401 27069 ts_tablet_manager.cc:1367] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:00.711594 27067 ts_tablet_manager.cc:1398] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:00.711819 27067 tablet_bootstrap.cc:492] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a: Bootstrap starting.
I20240502 07:59:00.711829 27069 raft_consensus.cc:348] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:59:00.711925 27069 raft_consensus.cc:374] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.711959 27069 raft_consensus.cc:729] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.712077 27069 consensus_queue.cc:260] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:59:00.712349 27069 ts_tablet_manager.cc:1398] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:00.712517 27069 tablet_bootstrap.cc:492] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:59:00.712996 27067 tablet_bootstrap.cc:654] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.713395 27069 tablet_bootstrap.cc:654] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.714517 27069 tablet_bootstrap.cc:492] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5: No bootstrap required, opened a new log
I20240502 07:59:00.714591 27069 ts_tablet_manager.cc:1367] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:00.714638 27064 tablet_bootstrap.cc:492] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:59:00.714713 27064 ts_tablet_manager.cc:1367] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:59:00.715047 27069 raft_consensus.cc:348] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:59:00.715152 27069 raft_consensus.cc:374] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.715183 27069 raft_consensus.cc:729] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.715119 27064 raft_consensus.cc:348] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:59:00.715227 27064 raft_consensus.cc:374] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.715260 27064 raft_consensus.cc:729] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.715387 27064 consensus_queue.cc:260] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:59:00.715790 27064 ts_tablet_manager.cc:1398] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:00.715852 27069 consensus_queue.cc:260] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:59:00.716041 27064 tablet_bootstrap.cc:492] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177: Bootstrap starting.
I20240502 07:59:00.716162 27069 ts_tablet_manager.cc:1398] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:00.716873 27064 tablet_bootstrap.cc:654] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:00.718075 27064 tablet_bootstrap.cc:492] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177: No bootstrap required, opened a new log
I20240502 07:59:00.718178 27064 ts_tablet_manager.cc:1367] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 07:59:00.718434 27067 tablet_bootstrap.cc:492] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a: No bootstrap required, opened a new log
I20240502 07:59:00.718528 27067 ts_tablet_manager.cc:1367] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 07:59:00.718780 27064 raft_consensus.cc:348] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:59:00.718880 27064 raft_consensus.cc:374] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.718926 27064 raft_consensus.cc:729] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.718966 27067 raft_consensus.cc:348] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:59:00.719055 27067 raft_consensus.cc:374] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:00.719085 27067 raft_consensus.cc:729] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Initialized, Role: FOLLOWER
I20240502 07:59:00.719049 27064 consensus_queue.cc:260] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:59:00.719225 27067 consensus_queue.cc:260] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:59:00.719506 27067 ts_tablet_manager.cc:1398] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:00.719830 27064 ts_tablet_manager.cc:1398] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:59:00.760563 27078 raft_consensus.cc:482] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:00.760691 27078 raft_consensus.cc:504] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:59:00.761090 27078 leader_election.cc:288] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:00.761462 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c81a7795712e45b5a7fd97450a9f06a7" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:59:00.761448 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c81a7795712e45b5a7fd97450a9f06a7" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" is_pre_election: true
I20240502 07:59:00.761590 25759 raft_consensus.cc:2454] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfa00c061c748b18772d5f0566944b5 in term 0.
I20240502 07:59:00.761595 25890 raft_consensus.cc:2454] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfa00c061c748b18772d5f0566944b5 in term 0.
I20240502 07:59:00.761884 25563 leader_election.cc:302] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, 14d8fd1fc3a04d9d967b8033df84f34a; no voters:
I20240502 07:59:00.762035 27078 raft_consensus.cc:2781] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:00.762095 27078 raft_consensus.cc:482] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:00.762133 27078 raft_consensus.cc:3037] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.762905 27078 raft_consensus.cc:504] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:59:00.763206 27078 leader_election.cc:288] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 election: Requested vote from peers e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865), 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:00.763492 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c81a7795712e45b5a7fd97450a9f06a7" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:59:00.763520 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c81a7795712e45b5a7fd97450a9f06a7" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
I20240502 07:59:00.763597 25890 raft_consensus.cc:3037] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.763612 25759 raft_consensus.cc:3037] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.764328 25890 raft_consensus.cc:2454] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfa00c061c748b18772d5f0566944b5 in term 1.
I20240502 07:59:00.764385 25759 raft_consensus.cc:2454] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfa00c061c748b18772d5f0566944b5 in term 1.
I20240502 07:59:00.764590 25564 leader_election.cc:302] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:59:00.764724 27078 raft_consensus.cc:2781] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:00.764839 27078 raft_consensus.cc:686] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Becoming Leader. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Running, Role: LEADER
I20240502 07:59:00.764977 27078 consensus_queue.cc:237] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }
I20240502 07:59:00.766636 25351 catalog_manager.cc:5484] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:00.800863 26712 raft_consensus.cc:482] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:00.800994 26712 raft_consensus.cc:504] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:59:00.801417 26712 leader_election.cc:288] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:59:00.801811 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843d1c1d6384476983749a78f65a5c74" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" is_pre_election: true
I20240502 07:59:00.801838 25627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843d1c1d6384476983749a78f65a5c74" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5" is_pre_election: true
I20240502 07:59:00.801957 25759 raft_consensus.cc:2454] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 0.
I20240502 07:59:00.801968 25627 raft_consensus.cc:2454] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 0.
I20240502 07:59:00.802266 25827 leader_election.cc:302] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:59:00.802434 26712 raft_consensus.cc:2781] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:00.802500 26712 raft_consensus.cc:482] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:00.802538 26712 raft_consensus.cc:3037] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.803495 26712 raft_consensus.cc:504] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:59:00.803893 26712 leader_election.cc:288] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 election: Requested vote from peers 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:59:00.804287 25759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843d1c1d6384476983749a78f65a5c74" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
I20240502 07:59:00.804287 25627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843d1c1d6384476983749a78f65a5c74" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:59:00.804400 25627 raft_consensus.cc:3037] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.804404 25759 raft_consensus.cc:3037] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.805261 25759 raft_consensus.cc:2454] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 1.
I20240502 07:59:00.805341 25627 raft_consensus.cc:2454] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 1.
I20240502 07:59:00.805588 25825 leader_election.cc:302] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:59:00.805763 26712 raft_consensus.cc:2781] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:00.805836 26712 raft_consensus.cc:686] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Becoming Leader. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Running, Role: LEADER
I20240502 07:59:00.805950 26712 consensus_queue.cc:237] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } }
I20240502 07:59:00.816346 25351 catalog_manager.cc:5484] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9596bfb8d324b4da93a4288771ed177 (127.18.13.132). New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:00.976480 27078 raft_consensus.cc:482] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:00.976617 27078 raft_consensus.cc:504] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:59:00.977075 27078 leader_election.cc:288] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:59:00.977536 25496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a8d5798c8943658113d67c88f4ad25" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" is_pre_election: true
I20240502 07:59:00.977557 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a8d5798c8943658113d67c88f4ad25" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177" is_pre_election: true
I20240502 07:59:00.977684 25890 raft_consensus.cc:2454] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfa00c061c748b18772d5f0566944b5 in term 0.
I20240502 07:59:00.977701 25496 raft_consensus.cc:2454] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfa00c061c748b18772d5f0566944b5 in term 0.
I20240502 07:59:00.977988 25564 leader_election.cc:302] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:59:00.978170 27078 raft_consensus.cc:2781] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:00.978232 27078 raft_consensus.cc:482] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:00.978261 27078 raft_consensus.cc:3037] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.979066 27078 raft_consensus.cc:504] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:59:00.979372 27078 leader_election.cc:288] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 election: Requested vote from peers c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:59:00.979758 25496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a8d5798c8943658113d67c88f4ad25" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
I20240502 07:59:00.979758 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21a8d5798c8943658113d67c88f4ad25" candidate_uuid: "0dfa00c061c748b18772d5f0566944b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:59:00.979908 25496 raft_consensus.cc:3037] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.979908 25890 raft_consensus.cc:3037] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:00.980871 25890 raft_consensus.cc:2454] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfa00c061c748b18772d5f0566944b5 in term 1.
I20240502 07:59:00.980871 25496 raft_consensus.cc:2454] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfa00c061c748b18772d5f0566944b5 in term 1.
I20240502 07:59:00.981156 25565 leader_election.cc:302] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:59:00.981304 27078 raft_consensus.cc:2781] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:00.981390 27078 raft_consensus.cc:686] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Becoming Leader. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Running, Role: LEADER
I20240502 07:59:00.981536 27078 consensus_queue.cc:237] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } }
I20240502 07:59:00.983364 25351 catalog_manager.cc:5484] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:01.117475 27131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:01.117771 27131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:01.121004 27131 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 07:59:01.194540 27078 consensus_queue.cc:1035] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:01.205425 26915 consensus_queue.cc:1035] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:01.214198 25890 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:01.214622 27078 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:01.214687 25496 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:01.215101 27078 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:01.219913 25627 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:01.220079 25760 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:01.222390 26712 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:01.227231 26712 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 132.158 ms
time per row: 0.0660792 ms
I20240502 07:59:01.297716 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.133:39097
--local_ip_for_outbound_sockets=127.18.13.133
--tserver_master_addrs=127.18.13.190:46035
--webserver_port=45141
--webserver_interface=127.18.13.133
--builtin_ntp_servers=127.18.13.148:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 07:59:01.412456 27194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:01.412709 27194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:01.412758 27194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:01.416005 27194 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:01.416090 27194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:01.416168 27194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.133
I20240502 07:59:01.420316 27194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.133:39097
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.18.13.133
--webserver_port=45141
--tserver_master_addrs=127.18.13.190:46035
--never_fsync=true
--heap_profile_path=/tmp/kudu.27194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.133
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:01.421270 27194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:01.422415 27194 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:01.431259 27202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:01.432260 27199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:01.433514 27200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:01.434852 27194 server_base.cc:1034] running on GCE node
I20240502 07:59:01.435326 27194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:01.436013 27194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:01.437228 27194 hybrid_clock.cc:648] HybridClock initialized: now 1714636741437202 us; error 49 us; skew 500 ppm
I20240502 07:59:01.444031 27194 webserver.cc:460] Webserver started at http://127.18.13.133:45141/ using document root <none> and password file <none>
I20240502 07:59:01.444773 27194 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:01.444876 27194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:01.449149 27194 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 07:59:01.451365 27208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:01.452559 27194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:01.452680 27194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c5f3003b08fb4214b116c0be8dd62851"
format_stamp: "Formatted at 2024-05-02 07:58:50 on dist-test-slave-f46z"
I20240502 07:59:01.453142 27194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:01.467033 27194 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:01.467707 27194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:01.467898 27194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:01.468441 27194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:01.469406 27194 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:01.469455 27194 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:01.469513 27194 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:01.469537 27194 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:01.478956 27194 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.133:39097
I20240502 07:59:01.478984 27321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.133:39097 every 8 connection(s)
I20240502 07:59:01.480141 27194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:59:01.484051 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 27194
I20240502 07:59:01.484603 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.134:43037
--local_ip_for_outbound_sockets=127.18.13.134
--tserver_master_addrs=127.18.13.190:46035
--webserver_port=42073
--webserver_interface=127.18.13.134
--builtin_ntp_servers=127.18.13.148:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:01.490902 27322 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46035
I20240502 07:59:01.491210 27322 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:01.492048 27322 heartbeater.cc:511] Master 127.18.13.190:46035 requested a full tablet report, sending...
I20240502 07:59:01.493439 25351 ts_manager.cc:194] Re-registered known tserver with Master: c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097)
I20240502 07:59:01.494073 25351 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.133:34735
W20240502 07:59:01.617630 27325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:01.617895 27325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:01.617941 27325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:01.621264 27325 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:01.621361 27325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:01.621436 27325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.134
I20240502 07:59:01.626040 27325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.134:43037
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.18.13.134
--webserver_port=42073
--tserver_master_addrs=127.18.13.190:46035
--never_fsync=true
--heap_profile_path=/tmp/kudu.27325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.134
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:01.627239 27325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:01.628791 27325 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:01.639647 27332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:01.640808 27334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:01.641263 27331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:01.642349 27325 server_base.cc:1034] running on GCE node
I20240502 07:59:01.642869 27325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:01.643610 27325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:01.644788 27325 hybrid_clock.cc:648] HybridClock initialized: now 1714636741644785 us; error 53 us; skew 500 ppm
I20240502 07:59:01.647217 27325 webserver.cc:460] Webserver started at http://127.18.13.134:42073/ using document root <none> and password file <none>
I20240502 07:59:01.647863 27325 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:01.647997 27325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:01.651578 27325 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:01.653723 27340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:01.654877 27325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 07:59:01.654985 27325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a17bbff6a71b4667b14f5dd12bca3057"
format_stamp: "Formatted at 2024-05-02 07:58:51 on dist-test-slave-f46z"
I20240502 07:59:01.655417 27325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:01.672816 27325 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:01.673483 27325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:01.673640 27325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:01.674300 27325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:01.675347 27325 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:01.675400 27325 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:01.675457 27325 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:01.675483 27325 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:01.684599 27325 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.134:43037
I20240502 07:59:01.684664 27453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.134:43037 every 8 connection(s)
I20240502 07:59:01.685691 27325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 07:59:01.691597 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 27325
I20240502 07:59:01.692070 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.135:34615
--local_ip_for_outbound_sockets=127.18.13.135
--tserver_master_addrs=127.18.13.190:46035
--webserver_port=33083
--webserver_interface=127.18.13.135
--builtin_ntp_servers=127.18.13.148:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:01.695108 27454 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46035
I20240502 07:59:01.695396 27454 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:01.696173 27454 heartbeater.cc:511] Master 127.18.13.190:46035 requested a full tablet report, sending...
I20240502 07:59:01.697417 25351 ts_manager.cc:194] Re-registered known tserver with Master: a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037)
I20240502 07:59:01.697975 25351 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.134:38493
W20240502 07:59:01.815941 27457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:01.816298 27457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:01.816376 27457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:01.820952 27457 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:01.821069 27457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:01.821185 27457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.135
I20240502 07:59:01.826568 27457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.135:34615
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.18.13.135
--webserver_port=33083
--tserver_master_addrs=127.18.13.190:46035
--never_fsync=true
--heap_profile_path=/tmp/kudu.27457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.135
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:01.827664 27457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:01.828895 27457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:01.838876 27463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:01.839656 27462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:01.840373 27465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:01.841542 27457 server_base.cc:1034] running on GCE node
I20240502 07:59:01.842044 27457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:01.842638 27457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:01.843806 27457 hybrid_clock.cc:648] HybridClock initialized: now 1714636741843767 us; error 41 us; skew 500 ppm
I20240502 07:59:01.846060 27457 webserver.cc:460] Webserver started at http://127.18.13.135:33083/ using document root <none> and password file <none>
I20240502 07:59:01.846660 27457 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:01.846730 27457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:01.850591 27457 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20240502 07:59:01.852795 27471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:01.853837 27457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:01.853952 27457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c6746058199446968d2496df55cdee42"
format_stamp: "Formatted at 2024-05-02 07:58:51 on dist-test-slave-f46z"
I20240502 07:59:01.854394 27457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:01.866355 27457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:01.867020 27457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:01.867183 27457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:01.867724 27457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:01.868711 27457 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:01.868768 27457 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:01.868839 27457 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:01.868868 27457 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:01.877683 27457 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.135:34615
I20240502 07:59:01.877705 27584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.135:34615 every 8 connection(s)
I20240502 07:59:01.878862 27457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 07:59:01.888516 27585 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46035
I20240502 07:59:01.888821 27585 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:01.889043 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 27457
I20240502 07:59:01.889528 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.136:38043
--local_ip_for_outbound_sockets=127.18.13.136
--tserver_master_addrs=127.18.13.190:46035
--webserver_port=40371
--webserver_interface=127.18.13.136
--builtin_ntp_servers=127.18.13.148:45871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:01.889586 27585 heartbeater.cc:511] Master 127.18.13.190:46035 requested a full tablet report, sending...
I20240502 07:59:01.890826 25351 ts_manager.cc:194] Re-registered known tserver with Master: c6746058199446968d2496df55cdee42 (127.18.13.135:34615)
I20240502 07:59:01.891487 25351 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.135:42387
W20240502 07:59:02.005357 27588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:02.005657 27588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:02.005731 27588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:02.009466 27588 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:02.009563 27588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:02.009678 27588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.136
I20240502 07:59:02.013576 27588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:45871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.136:38043
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.18.13.136
--webserver_port=40371
--tserver_master_addrs=127.18.13.190:46035
--never_fsync=true
--heap_profile_path=/tmp/kudu.27588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.136
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:02.014549 27588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:02.015724 27588 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:02.025023 27594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:02.025408 27597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:02.026732 27595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:02.027995 27588 server_base.cc:1034] running on GCE node
I20240502 07:59:02.028508 27588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:02.029093 27588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:02.030267 27588 hybrid_clock.cc:648] HybridClock initialized: now 1714636742030237 us; error 47 us; skew 500 ppm
I20240502 07:59:02.032363 27588 webserver.cc:460] Webserver started at http://127.18.13.136:40371/ using document root <none> and password file <none>
I20240502 07:59:02.032925 27588 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:02.033003 27588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:02.036448 27588 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:02.038393 27603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:02.039423 27588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:02.039531 27588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "85fbc6c98f0643469c92215a5cfb2e38"
format_stamp: "Formatted at 2024-05-02 07:58:51 on dist-test-slave-f46z"
I20240502 07:59:02.040099 27588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:02.057466 27588 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:02.058072 27588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:02.058216 27588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:02.058728 27588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:02.059633 27588 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:02.059682 27588 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:02.059756 27588 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:02.059813 27588 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:02.068122 27588 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.136:38043
I20240502 07:59:02.068153 27716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.136:38043 every 8 connection(s)
I20240502 07:59:02.069272 27588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 07:59:02.076284 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 27588
I20240502 07:59:02.079226 27717 heartbeater.cc:347] Connected to a master server at 127.18.13.190:46035
I20240502 07:59:02.079514 27717 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:02.080302 27717 heartbeater.cc:511] Master 127.18.13.190:46035 requested a full tablet report, sending...
I20240502 07:59:02.081733 25351 ts_manager.cc:194] Re-registered known tserver with Master: 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043)
I20240502 07:59:02.082319 25351 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.136:32925
I20240502 07:59:02.494699 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.496861 27322 heartbeater.cc:503] Master 127.18.13.190:46035 was elected leader, sending a full tablet report...
I20240502 07:59:02.498147 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:02.504451 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.510056 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:02.515483 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.537539 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:02.556674 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:02.558291 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.564218 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.588770 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:02.602038 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.612200 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:02.612638 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:02.629877 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.632522 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:02.662981 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:02.666342 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.671707 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.672250 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.680303 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:02.685916 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.686220 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:02.701371 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:02.701682 27454 heartbeater.cc:503] Master 127.18.13.190:46035 was elected leader, sending a full tablet report...
I20240502 07:59:02.703907 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:02.709812 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:02.726226 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.734519 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.738842 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.739073 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.745466 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.759732 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.774333 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:02.778360 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.794087 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:02.799705 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:02.800180 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.802356 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:02.803189 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.841369 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:02.848927 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:02.894773 27585 heartbeater.cc:503] Master 127.18.13.190:46035 was elected leader, sending a full tablet report...
I20240502 07:59:02.987599 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.000444 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:03.002311 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:03.006214 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:03.013223 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.021270 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.028895 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:03.051824 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.076121 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:03.088014 27717 heartbeater.cc:503] Master 127.18.13.190:46035 was elected leader, sending a full tablet report...
I20240502 07:59:03.109148 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:03.137115 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:03.138619 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:03.146633 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.147670 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.161975 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.163203 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:03.163707 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:03.165843 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.175426 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.180106 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:03.184073 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.198197 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.201326 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.207226 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:03.323942 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.326575 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.333276 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:03.338745 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:03.352700 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:03.352839 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.354049 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:03.381335 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:03.385152 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.390262 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.397228 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:03.409883 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.411680 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:03.414777 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.415937 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:03.424649 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.520826 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:03.533414 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.535755 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:03.536176 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:03.548458 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.564635 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.576759 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:03.592988 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.620397 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:03.635233 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.641288 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.641367 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.650220 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:03.652006 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:03.655815 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.670162 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:03.723218 25497 consensus_queue.cc:237] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:03.725322 25760 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:03.725808 26968 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:03.727722 25891 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:03.728765 26968 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:59:03.737818 25433 consensus_peers.cc:485] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1fa713b432740d98bcab918d251b54d. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:03.739461 28374 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:03.742451 25760 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:03.746685 25349 catalog_manager.cc:5484] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:03.750082 25891 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:03.758879 25891 consensus_queue.cc:237] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:03.760609 25629 raft_consensus.cc:1261] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:03.761314 25497 raft_consensus.cc:1261] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:03.761529 28393 consensus_queue.cc:1035] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:03.763571 28393 consensus_queue.cc:1035] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:03.766486 26712 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:03.767387 25497 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:03.776314 25629 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:03.779045 25497 consensus_queue.cc:237] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
W20240502 07:59:03.780964 25824 consensus_peers.cc:485] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6929d2560404acb9bb53f54a181f958. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:03.780858 25350 catalog_manager.cc:5484] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:03.781682 25891 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:03.781685 25760 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:03.782099 28374 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:03.782313 26968 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:59:03.789160 25431 consensus_peers.cc:485] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff31ddd6b3d74a9aae6617afb56c40f5. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:03.792587 26968 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:03.795053 25760 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:03.796005 25347 catalog_manager.cc:5484] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:03.802191 25890 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:03.804944 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:03.819411 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.828142 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.828258 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:03.831409 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.833853 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:03.838575 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:03.852732 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:03.860435 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.869975 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.880292 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:03.890008 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:03.917407 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.920483 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:03.927412 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.932904 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:03.945474 25760 consensus_queue.cc:237] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:03.947387 25629 raft_consensus.cc:1261] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:03.947863 25891 raft_consensus.cc:1261] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:03.948447 27589 consensus_queue.cc:1035] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:03.949128 27589 consensus_queue.cc:1035] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:59:03.955540 25694 consensus_peers.cc:485] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71a67cb0f6b54bfb9eb44400fe21ce40. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:03.962325 27589 raft_consensus.cc:2932] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:03.962939 25629 raft_consensus.cc:2932] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:03.967258 25347 catalog_manager.cc:5484] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:03.968468 25891 raft_consensus.cc:2932] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.003962 25497 consensus_queue.cc:237] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:04.005894 25891 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.006495 25629 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.007404 28374 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.008265 28374 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:59:04.012360 25432 consensus_peers.cc:485] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 056ba8d15d18409e946c905b8876097a. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.019618 28374 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.021589 25629 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.023418 25353 catalog_manager.cc:5484] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.025251 25891 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.035771 25497 consensus_queue.cc:237] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:04.038096 25629 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.038573 26968 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.039289 25760 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.041705 28375 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:59:04.043206 25434 consensus_peers.cc:485] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb0c33c3e4014eb3b931d64adac60e1a. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.045097 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.051893 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:04.051980 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:04.064067 28375 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.066622 25629 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.069533 25347 catalog_manager.cc:5484] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.075517 25760 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.089087 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.089556 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:04.090842 25629 consensus_queue.cc:237] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:04.093909 25760 consensus_queue.cc:237] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:04.094173 25497 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.095557 25891 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.096146 25497 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.096112 27181 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.096900 28479 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.098659 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.098817 27181 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:59:04.100883 25695 consensus_peers.cc:485] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9722dae4e4414084bdd8066b41bcc47a. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.101493 25627 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.101891 28479 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.104696 28479 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.107865 25350 catalog_manager.cc:5484] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.108027 25496 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.112416 27181 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.114297 25497 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.118705 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.120496 25350 catalog_manager.cc:5484] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.123466 25627 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.142288 25890 consensus_queue.cc:237] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:04.144105 25760 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:59:04.144603 25824 consensus_peers.cc:485] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0aa60608089c4f50bcee339d01815a65. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.144781 26712 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.145565 25497 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.146049 26712 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.147265 25891 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
W20240502 07:59:04.148949 25564 consensus_peers.cc:485] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21a8d5798c8943658113d67c88f4ad25. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.149364 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:04.151522 28395 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.153158 25496 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.154305 25497 consensus_queue.cc:237] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:04.155287 25760 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.155992 25759 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:59:04.156810 25431 consensus_peers.cc:485] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a0605c8c64f4c6fb5e7315a118a0d7f. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.156980 25347 catalog_manager.cc:5484] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.157428 25627 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.157840 28375 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.160835 28375 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.162927 26968 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.164304 25760 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.167126 25627 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.169958 25497 consensus_queue.cc:237] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:04.171833 25627 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.171908 25891 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.172566 28515 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 07:59:04.172765 25433 consensus_peers.cc:485] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 056ba8d15d18409e946c905b8876097a. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.172859 28515 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.172870 25356 catalog_manager.cc:5484] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.174890 28515 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
W20240502 07:59:04.176075 25432 consensus_peers.cc:485] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 056ba8d15d18409e946c905b8876097a. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.178229 25891 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.181566 25347 catalog_manager.cc:5484] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.181703 25627 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.194087 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:04.196424 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.201197 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:04.207185 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.209334 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.209440 28586 ts_tablet_manager.cc:902] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:04.210318 28586 tablet_copy_client.cc:308] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:04.238978 25518 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet ff31ddd6b3d74a9aae6617afb56c40f5 from peer a17bbff6a71b4667b14f5dd12bca3057 ({username='slave'} at 127.18.13.134:48333)
I20240502 07:59:04.239181 25518 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet ff31ddd6b3d74a9aae6617afb56c40f5 from peer a17bbff6a71b4667b14f5dd12bca3057 at {username='slave'} at 127.18.13.134:48333: session id = a17bbff6a71b4667b14f5dd12bca3057-ff31ddd6b3d74a9aae6617afb56c40f5
I20240502 07:59:04.240749 25518 tablet_copy_source_session.cc:205] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.242048 25497 consensus_queue.cc:237] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:04.243842 25891 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.245033 28374 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.247067 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 07:59:04.247859 25432 consensus_peers.cc:485] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee25634bf534443b90fdbdcebdbcb02e. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.248831 25627 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.251679 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.251865 28586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff31ddd6b3d74a9aae6617afb56c40f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.252584 28374 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.258682 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:04.268399 28374 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.272303 25350 catalog_manager.cc:5484] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.275081 25891 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.275550 28586 tablet_copy_client.cc:783] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.275894 28586 tablet_copy_client.cc:647] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.280236 28586 tablet_copy_client.cc:523] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:04.284478 28586 tablet_bootstrap.cc:492] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap starting.
I20240502 07:59:04.290701 25760 consensus_queue.cc:237] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:04.292791 25629 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.293417 25891 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.293406 28586 log.cc:824] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:04.293674 28465 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.294312 28465 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:59:04.296861 25696 consensus_peers.cc:485] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87d6266008784883891ca32c01363dbe. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.299450 25627 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.299876 28465 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.303700 25497 consensus_queue.cc:237] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:04.304709 25891 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
W20240502 07:59:04.307608 25434 consensus_peers.cc:485] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd68090fdadf45faa0e3b2936b30047e. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.308421 25347 catalog_manager.cc:5484] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.314965 25759 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.316707 25891 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.317221 28515 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.317662 28515 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.319111 25760 consensus_queue.cc:237] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:04.319902 25627 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
W20240502 07:59:04.321293 25695 consensus_peers.cc:485] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9722dae4e4414084bdd8066b41bcc47a. This is attempt 1: this message will repeat every 5th retry.
W20240502 07:59:04.326920 25693 consensus_peers.cc:485] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9722dae4e4414084bdd8066b41bcc47a. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.327072 25497 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.327569 28464 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.329448 25627 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.329893 28464 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.330034 28479 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.331140 25629 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.331369 28586 tablet_bootstrap.cc:492] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:04.331724 28586 tablet_bootstrap.cc:492] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap complete.
I20240502 07:59:04.332091 28586 ts_tablet_manager.cc:1367] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057: Time spent bootstrapping tablet: real 0.048s user 0.035s sys 0.004s
I20240502 07:59:04.331954 25497 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.334918 25347 catalog_manager.cc:5484] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.336463 28586 raft_consensus.cc:348] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:04.336751 28586 raft_consensus.cc:729] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a17bbff6a71b4667b14f5dd12bca3057, State: Initialized, Role: LEARNER
I20240502 07:59:04.337167 28586 consensus_queue.cc:260] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:04.338212 28586 ts_tablet_manager.cc:1398] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240502 07:59:04.339071 25518 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session a17bbff6a71b4667b14f5dd12bca3057-ff31ddd6b3d74a9aae6617afb56c40f5 received from {username='slave'} at 127.18.13.134:48333
I20240502 07:59:04.339174 25518 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session a17bbff6a71b4667b14f5dd12bca3057-ff31ddd6b3d74a9aae6617afb56c40f5 on tablet ff31ddd6b3d74a9aae6617afb56c40f5 with peer a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:04.345871 26968 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.348805 25497 consensus_queue.cc:237] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
W20240502 07:59:04.351027 25434 consensus_peers.cc:485] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5eb2de2942914800bdbe924d45a9b1e7. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.351718 25891 raft_consensus.cc:1261] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.352671 25758 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.352952 28643 ts_tablet_manager.cc:902] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:04.353894 28643 tablet_copy_client.cc:308] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:04.354615 25891 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.357415 25349 catalog_manager.cc:5484] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.359035 28375 consensus_queue.cc:1035] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.360064 25760 raft_consensus.cc:1261] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.360921 28515 consensus_queue.cc:1035] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.361418 25497 consensus_queue.cc:237] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:04.363682 25518 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet a1fa713b432740d98bcab918d251b54d from peer c6746058199446968d2496df55cdee42 ({username='slave'} at 127.18.13.135:42665)
I20240502 07:59:04.364019 28586 ts_tablet_manager.cc:902] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:59:04.364018 25759 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.364549 28586 tablet_copy_client.cc:308] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.132:45865
I20240502 07:59:04.364701 25889 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.365164 25518 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet a1fa713b432740d98bcab918d251b54d from peer c6746058199446968d2496df55cdee42 at {username='slave'} at 127.18.13.135:42665: session id = c6746058199446968d2496df55cdee42-a1fa713b432740d98bcab918d251b54d
I20240502 07:59:04.365456 28375 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.366235 25518 tablet_copy_source_session.cc:205] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 07:59:04.370141 25433 consensus_peers.cc:485] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1fa713b432740d98bcab918d251b54d. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.370254 28375 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.370474 28643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fa713b432740d98bcab918d251b54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.370437 26968 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.371254 25760 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
W20240502 07:59:04.374068 25434 consensus_peers.cc:485] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1fa713b432740d98bcab918d251b54d. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.374737 28374 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.375779 25627 consensus_queue.cc:237] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:04.376025 25911 tablet_copy_service.cc:140] P e9596bfb8d324b4da93a4288771ed177: Received BeginTabletCopySession request for tablet d6929d2560404acb9bb53f54a181f958 from peer a17bbff6a71b4667b14f5dd12bca3057 ({username='slave'} at 127.18.13.134:49677)
I20240502 07:59:04.376180 25911 tablet_copy_service.cc:161] P e9596bfb8d324b4da93a4288771ed177: Beginning new tablet copy session on tablet d6929d2560404acb9bb53f54a181f958 from peer a17bbff6a71b4667b14f5dd12bca3057 at {username='slave'} at 127.18.13.134:49677: session id = a17bbff6a71b4667b14f5dd12bca3057-d6929d2560404acb9bb53f54a181f958
I20240502 07:59:04.377741 25911 tablet_copy_source_session.cc:205] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.379015 28586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6929d2560404acb9bb53f54a181f958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.379045 25349 catalog_manager.cc:5484] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.379686 25889 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.380167 25890 raft_consensus.cc:1261] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.381731 28461 consensus_queue.cc:1035] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.382405 25759 raft_consensus.cc:1261] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:59:04.382576 25563 consensus_peers.cc:485] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c81a7795712e45b5a7fd97450a9f06a7. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.382912 28586 tablet_copy_client.cc:783] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.383145 28586 tablet_copy_client.cc:647] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.383272 28461 consensus_queue.cc:1035] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.384023 25760 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.385422 25347 catalog_manager.cc:5484] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.391173 25758 consensus_queue.cc:237] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:04.391235 28461 raft_consensus.cc:2932] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.393072 25627 raft_consensus.cc:1261] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:59:04.393926 25695 consensus_peers.cc:485] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5958bb5c8fc4c1d91ae8f1e8b3861d0. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.393968 27589 consensus_queue.cc:1035] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.394042 25891 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.395077 25356 catalog_manager.cc:5484] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.395501 25890 raft_consensus.cc:2932] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.400265 28586 tablet_copy_client.cc:523] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:04.402137 28586 tablet_bootstrap.cc:492] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap starting.
I20240502 07:59:04.404578 25889 raft_consensus.cc:1261] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.405089 27589 consensus_queue.cc:1035] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.407218 25497 consensus_queue.cc:237] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:04.409387 27408 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.410521 28515 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 07:59:04.414469 25433 consensus_peers.cc:485] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff31ddd6b3d74a9aae6617afb56c40f5. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.415083 25758 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.415652 28675 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.416059 28464 raft_consensus.cc:2932] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.418954 25889 raft_consensus.cc:2932] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.419193 28515 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.422459 25758 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.422941 25355 catalog_manager.cc:5484] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.423276 25350 catalog_manager.cc:5484] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.425230 28643 tablet_copy_client.cc:783] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.425482 28643 tablet_copy_client.cc:647] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.425977 25889 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.426645 28375 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.427171 28643 tablet_copy_client.cc:523] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:04.427940 25890 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.433112 25627 raft_consensus.cc:2932] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.441097 28643 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Bootstrap starting.
I20240502 07:59:04.441385 25497 consensus_queue.cc:237] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:04.441726 25759 raft_consensus.cc:2932] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
W20240502 07:59:04.443305 25432 consensus_peers.cc:485] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee25634bf534443b90fdbdcebdbcb02e. This is attempt 1: this message will repeat every 5th retry.
W20240502 07:59:04.444036 25433 consensus_peers.cc:485] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee25634bf534443b90fdbdcebdbcb02e. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.444748 25627 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.445161 28375 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.446960 28375 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.450558 25627 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.453517 25889 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.453995 25347 catalog_manager.cc:5484] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.454457 28515 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.455391 25890 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.464164 28705 ts_tablet_manager.cc:902] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:04.466933 28705 tablet_copy_client.cc:308] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:04.468849 25497 consensus_queue.cc:237] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
W20240502 07:59:04.471532 25432 consensus_peers.cc:485] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8904fa47fc274c14acd291b4c91964be. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.472435 25759 consensus_queue.cc:237] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:04.473762 25889 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.474126 25890 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:59:04.474174 25694 consensus_peers.cc:485] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 273e203d4ed44c43859f08e9bec08a9d. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.474409 25627 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.474570 28479 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.474645 25629 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.474772 28515 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.475004 28479 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.475154 28515 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.481917 28643 log.cc:824] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:04.482841 28515 raft_consensus.cc:1050] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 to VOTER
I20240502 07:59:04.483412 28515 consensus_queue.cc:237] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:04.483507 27408 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
W20240502 07:59:04.486274 25433 consensus_peers.cc:485] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff31ddd6b3d74a9aae6617afb56c40f5. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.486629 28479 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.488479 25889 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:04.488844 28375 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:04.489913 25759 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:04.490607 25629 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.491405 25354 catalog_manager.cc:5484] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.492475 28586 tablet_bootstrap.cc:492] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:04.492588 27407 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:04.492810 28586 tablet_bootstrap.cc:492] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap complete.
I20240502 07:59:04.492993 28586 ts_tablet_manager.cc:1367] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057: Time spent bootstrapping tablet: real 0.091s user 0.026s sys 0.003s
I20240502 07:59:04.493623 28586 raft_consensus.cc:348] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:04.493749 28586 raft_consensus.cc:729] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a17bbff6a71b4667b14f5dd12bca3057, State: Initialized, Role: LEARNER
I20240502 07:59:04.494664 28586 consensus_queue.cc:260] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:04.494946 26968 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:04.495090 28586 ts_tablet_manager.cc:1398] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:04.495131 28675 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:04.495685 25911 tablet_copy_service.cc:342] P e9596bfb8d324b4da93a4288771ed177: Request end of tablet copy session a17bbff6a71b4667b14f5dd12bca3057-d6929d2560404acb9bb53f54a181f958 received from {username='slave'} at 127.18.13.134:49677
I20240502 07:59:04.495782 25911 tablet_copy_service.cc:433] P e9596bfb8d324b4da93a4288771ed177: ending tablet copy session a17bbff6a71b4667b14f5dd12bca3057-d6929d2560404acb9bb53f54a181f958 on tablet d6929d2560404acb9bb53f54a181f958 with peer a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:04.497227 28375 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.498482 28675 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.499341 25759 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.500933 25891 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.501576 25780 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet 71a67cb0f6b54bfb9eb44400fe21ce40 from peer 85fbc6c98f0643469c92215a5cfb2e38 ({username='slave'} at 127.18.13.136:56619)
I20240502 07:59:04.501724 25780 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet 71a67cb0f6b54bfb9eb44400fe21ce40 from peer 85fbc6c98f0643469c92215a5cfb2e38 at {username='slave'} at 127.18.13.136:56619: session id = 85fbc6c98f0643469c92215a5cfb2e38-71a67cb0f6b54bfb9eb44400fe21ce40
I20240502 07:59:04.503242 25780 tablet_copy_source_session.cc:205] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.505044 25891 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.505492 25354 catalog_manager.cc:5484] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.506986 25628 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.509317 28705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a67cb0f6b54bfb9eb44400fe21ce40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.510183 25354 catalog_manager.cc:5484] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.510298 27407 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.512574 25890 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.521765 25497 consensus_queue.cc:237] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:04.523489 25890 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:04.524009 28374 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:04.526206 25759 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:04.526264 27407 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:04.526669 28374 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:04.526988 28374 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:04.529279 28374 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:04.531268 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ff31ddd6b3d74a9aae6617afb56c40f5 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:04.531746 25890 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:04.532984 25759 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:04.533089 25350 catalog_manager.cc:5484] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.534621 25628 consensus_queue.cc:237] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:04.536746 25759 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:59:04.536770 25564 consensus_peers.cc:485] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ad22d2e464d4defab6bd0061e6058f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.536890 25497 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.537197 27181 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.537395 28545 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.535916 28705 tablet_copy_client.cc:783] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.538908 28705 tablet_copy_client.cc:647] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.539242 27407 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:04.544699 28705 tablet_copy_client.cc:523] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:04.547513 25890 consensus_queue.cc:237] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:04.547655 27519 tablet_service.cc:1514] Processing DeleteTablet for tablet ff31ddd6b3d74a9aae6617afb56c40f5 with delete_type TABLET_DATA_TOMBSTONED (TS c6746058199446968d2496df55cdee42 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56612
I20240502 07:59:04.549944 25496 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.550339 25759 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.550746 28393 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 07:59:04.551023 25824 consensus_peers.cc:485] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0aa60608089c4f50bcee339d01815a65. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.551280 28393 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.551404 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:04.552554 28393 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.553045 28545 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.553448 28705 tablet_bootstrap.cc:492] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap starting.
I20240502 07:59:04.554591 25760 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.555709 25496 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.556393 25758 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.557289 25350 catalog_manager.cc:5484] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.559302 25496 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.559422 25347 catalog_manager.cc:5484] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.568408 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.570017 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 07:59:04.574568 25827 consensus_peers.cc:485] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0aa60608089c4f50bcee339d01815a65. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.578876 25496 consensus_queue.cc:237] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:04.580560 25889 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.581004 25760 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.581250 26968 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.581728 28375 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 07:59:04.582481 25434 consensus_peers.cc:485] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd68090fdadf45faa0e3b2936b30047e. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.584257 28515 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.588090 25758 consensus_queue.cc:237] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
W20240502 07:59:04.591020 25431 consensus_peers.cc:485] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd68090fdadf45faa0e3b2936b30047e. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.590930 25350 catalog_manager.cc:5484] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:04.591714 25696 consensus_peers.cc:485] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87d6266008784883891ca32c01363dbe. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.592118 25760 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.594344 25889 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.595746 28705 log.cc:824] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:04.598457 25890 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.598873 28465 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.600044 25628 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.600646 28465 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 07:59:04.601085 25693 consensus_peers.cc:485] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87d6266008784883891ca32c01363dbe. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.602876 27589 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.604055 25628 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.604497 28643 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 07:59:04.605005 28643 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Bootstrap complete.
I20240502 07:59:04.605337 28643 ts_tablet_manager.cc:1367] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Time spent bootstrapping tablet: real 0.164s user 0.040s sys 0.000s
I20240502 07:59:04.606739 25890 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.607581 25354 catalog_manager.cc:5484] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:04.608515 28643 raft_consensus.cc:348] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:04.609699 28643 raft_consensus.cc:729] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6746058199446968d2496df55cdee42, State: Initialized, Role: LEARNER
I20240502 07:59:04.610102 28643 consensus_queue.cc:260] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:04.611304 28643 ts_tablet_manager.cc:1398] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 07:59:04.612166 25518 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session c6746058199446968d2496df55cdee42-a1fa713b432740d98bcab918d251b54d received from {username='slave'} at 127.18.13.135:42665
I20240502 07:59:04.612248 25518 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session c6746058199446968d2496df55cdee42-a1fa713b432740d98bcab918d251b54d on tablet a1fa713b432740d98bcab918d251b54d with peer c6746058199446968d2496df55cdee42
I20240502 07:59:04.615033 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.626570 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:04.628332 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.629555 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:04.635390 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.645920 28643 ts_tablet_manager.cc:902] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42: Initiating tablet copy from peer 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:59:04.646401 28643 tablet_copy_client.cc:308] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:38825
W20240502 07:59:04.648591 25342 catalog_manager.cc:4797] TS c6746058199446968d2496df55cdee42 (127.18.13.135:34615): delete failed for tablet ff31ddd6b3d74a9aae6617afb56c40f5 because the tablet was not found. No further retry: Not found: Tablet not found: ff31ddd6b3d74a9aae6617afb56c40f5
I20240502 07:59:04.653681 28586 ts_tablet_manager.cc:902] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:04.654179 28586 tablet_copy_client.cc:308] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:04.656204 25518 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet 5a0605c8c64f4c6fb5e7315a118a0d7f from peer a17bbff6a71b4667b14f5dd12bca3057 ({username='slave'} at 127.18.13.134:48333)
I20240502 07:59:04.656350 25518 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet 5a0605c8c64f4c6fb5e7315a118a0d7f from peer a17bbff6a71b4667b14f5dd12bca3057 at {username='slave'} at 127.18.13.134:48333: session id = a17bbff6a71b4667b14f5dd12bca3057-5a0605c8c64f4c6fb5e7315a118a0d7f
I20240502 07:59:04.656373 25649 tablet_copy_service.cc:140] P 0dfa00c061c748b18772d5f0566944b5: Received BeginTabletCopySession request for tablet 21a8d5798c8943658113d67c88f4ad25 from peer c6746058199446968d2496df55cdee42 ({username='slave'} at 127.18.13.135:56049)
I20240502 07:59:04.656504 25649 tablet_copy_service.cc:161] P 0dfa00c061c748b18772d5f0566944b5: Beginning new tablet copy session on tablet 21a8d5798c8943658113d67c88f4ad25 from peer c6746058199446968d2496df55cdee42 at {username='slave'} at 127.18.13.135:56049: session id = c6746058199446968d2496df55cdee42-21a8d5798c8943658113d67c88f4ad25
I20240502 07:59:04.657434 25518 tablet_copy_source_session.cc:205] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.657986 25649 tablet_copy_source_session.cc:205] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.658541 28586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a0605c8c64f4c6fb5e7315a118a0d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.659596 28643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a8d5798c8943658113d67c88f4ad25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.661474 28586 tablet_copy_client.cc:783] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.661641 28586 tablet_copy_client.cc:647] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.661972 28643 tablet_copy_client.cc:783] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.662122 28643 tablet_copy_client.cc:647] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.662249 28776 ts_tablet_manager.cc:902] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:04.663100 28586 tablet_copy_client.cc:523] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:04.663421 28776 tablet_copy_client.cc:308] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:04.664937 28586 tablet_bootstrap.cc:492] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap starting.
I20240502 07:59:04.671321 28797 ts_tablet_manager.cc:902] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:04.672032 28797 tablet_copy_client.cc:308] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:04.674290 28643 tablet_copy_client.cc:523] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:04.675889 28643 tablet_bootstrap.cc:492] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42: Bootstrap starting.
I20240502 07:59:04.677022 25518 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet 056ba8d15d18409e946c905b8876097a from peer 85fbc6c98f0643469c92215a5cfb2e38 ({username='slave'} at 127.18.13.136:58689)
I20240502 07:59:04.677146 25518 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet 056ba8d15d18409e946c905b8876097a from peer 85fbc6c98f0643469c92215a5cfb2e38 at {username='slave'} at 127.18.13.136:58689: session id = 85fbc6c98f0643469c92215a5cfb2e38-056ba8d15d18409e946c905b8876097a
I20240502 07:59:04.678162 25518 tablet_copy_source_session.cc:205] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.693121 28820 ts_tablet_manager.cc:902] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:04.693840 28820 tablet_copy_client.cc:308] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:04.694516 25518 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet 056ba8d15d18409e946c905b8876097a from peer c6746058199446968d2496df55cdee42 ({username='slave'} at 127.18.13.135:42665)
I20240502 07:59:04.694635 25518 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet 056ba8d15d18409e946c905b8876097a from peer c6746058199446968d2496df55cdee42 at {username='slave'} at 127.18.13.135:42665: session id = c6746058199446968d2496df55cdee42-056ba8d15d18409e946c905b8876097a
I20240502 07:59:04.695607 25518 tablet_copy_source_session.cc:205] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.696955 28820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056ba8d15d18409e946c905b8876097a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.702309 25628 consensus_queue.cc:237] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:04.703333 25758 consensus_queue.cc:237] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:04.703924 28820 tablet_copy_client.cc:783] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.704134 28820 tablet_copy_client.cc:647] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.705006 25890 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.705446 25496 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:59:04.705890 25696 consensus_peers.cc:485] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03016ecfa11d4987a2e9d2368a66a5a2. This is attempt 1: this message will repeat every 5th retry.
W20240502 07:59:04.705979 25564 consensus_peers.cc:485] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240502 07:59:04.705979 25563 consensus_peers.cc:485] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21a8d5798c8943658113d67c88f4ad25. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.706095 28465 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.706208 28461 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.707098 25497 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.707566 28461 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.708418 28797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 056ba8d15d18409e946c905b8876097a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.709298 28820 tablet_copy_client.cc:523] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:04.710166 28461 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.711771 28820 tablet_bootstrap.cc:492] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42: Bootstrap starting.
I20240502 07:59:04.713162 25890 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.713901 25348 catalog_manager.cc:5484] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 4 to 5, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.717299 25496 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.719477 25628 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.720180 28464 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.720860 28586 tablet_bootstrap.cc:492] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:04.721175 28586 tablet_bootstrap.cc:492] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap complete.
I20240502 07:59:04.721380 28586 ts_tablet_manager.cc:1367] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057: Time spent bootstrapping tablet: real 0.057s user 0.029s sys 0.001s
I20240502 07:59:04.721495 28464 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.721912 28586 raft_consensus.cc:348] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:04.722040 28586 raft_consensus.cc:729] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a17bbff6a71b4667b14f5dd12bca3057, State: Initialized, Role: LEARNER
I20240502 07:59:04.722151 28586 consensus_queue.cc:260] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:04.724565 28586 ts_tablet_manager.cc:1398] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:04.724570 25496 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.724907 25350 catalog_manager.cc:5484] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.725091 25518 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session a17bbff6a71b4667b14f5dd12bca3057-5a0605c8c64f4c6fb5e7315a118a0d7f received from {username='slave'} at 127.18.13.134:48333
I20240502 07:59:04.725172 25518 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session a17bbff6a71b4667b14f5dd12bca3057-5a0605c8c64f4c6fb5e7315a118a0d7f on tablet 5a0605c8c64f4c6fb5e7315a118a0d7f with peer a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:04.732009 25518 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet eb0c33c3e4014eb3b931d64adac60e1a from peer c5f3003b08fb4214b116c0be8dd62851 ({username='slave'} at 127.18.13.133:46279)
I20240502 07:59:04.732156 25518 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet eb0c33c3e4014eb3b931d64adac60e1a from peer c5f3003b08fb4214b116c0be8dd62851 at {username='slave'} at 127.18.13.133:46279: session id = c5f3003b08fb4214b116c0be8dd62851-eb0c33c3e4014eb3b931d64adac60e1a
I20240502 07:59:04.733079 25518 tablet_copy_source_session.cc:205] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.734336 25628 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.737053 28776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0c33c3e4014eb3b931d64adac60e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.738225 25758 consensus_queue.cc:237] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:04.740520 25890 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.741286 25628 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:04.741593 28479 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:59:04.742084 25696 consensus_peers.cc:485] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3bc416015dc40d88743215d2cf46230. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.742218 28465 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:04.742506 28797 tablet_copy_client.cc:783] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.742707 28797 tablet_copy_client.cc:647] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.742744 28705 tablet_bootstrap.cc:492] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:04.743162 28705 tablet_bootstrap.cc:492] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap complete.
I20240502 07:59:04.743518 28705 ts_tablet_manager.cc:1367] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent bootstrapping tablet: real 0.190s user 0.033s sys 0.005s
I20240502 07:59:04.746264 28705 raft_consensus.cc:348] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:04.746546 28705 raft_consensus.cc:729] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85fbc6c98f0643469c92215a5cfb2e38, State: Initialized, Role: LEARNER
I20240502 07:59:04.746915 28705 consensus_queue.cc:260] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:04.747833 28797 tablet_copy_client.cc:523] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:04.747913 28705 ts_tablet_manager.cc:1398] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240502 07:59:04.748831 25780 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-71a67cb0f6b54bfb9eb44400fe21ce40 received from {username='slave'} at 127.18.13.136:56619
I20240502 07:59:04.748929 25780 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-71a67cb0f6b54bfb9eb44400fe21ce40 on tablet 71a67cb0f6b54bfb9eb44400fe21ce40 with peer 85fbc6c98f0643469c92215a5cfb2e38
I20240502 07:59:04.750576 28776 tablet_copy_client.cc:783] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.750865 28776 tablet_copy_client.cc:647] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.751636 28797 tablet_bootstrap.cc:492] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap starting.
I20240502 07:59:04.752777 28776 tablet_copy_client.cc:523] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:04.753751 28643 tablet_bootstrap.cc:492] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:04.754096 28643 tablet_bootstrap.cc:492] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42: Bootstrap complete.
I20240502 07:59:04.754276 28643 ts_tablet_manager.cc:1367] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42: Time spent bootstrapping tablet: real 0.078s user 0.029s sys 0.000s
I20240502 07:59:04.754693 28776 tablet_bootstrap.cc:492] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851: Bootstrap starting.
I20240502 07:59:04.754863 28643 raft_consensus.cc:348] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:04.755822 28643 raft_consensus.cc:729] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6746058199446968d2496df55cdee42, State: Initialized, Role: LEARNER
I20240502 07:59:04.755960 28643 consensus_queue.cc:260] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:04.756247 28643 ts_tablet_manager.cc:1398] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:04.756922 25649 tablet_copy_service.cc:342] P 0dfa00c061c748b18772d5f0566944b5: Request end of tablet copy session c6746058199446968d2496df55cdee42-21a8d5798c8943658113d67c88f4ad25 received from {username='slave'} at 127.18.13.135:56049
I20240502 07:59:04.757014 25649 tablet_copy_service.cc:433] P 0dfa00c061c748b18772d5f0566944b5: ending tablet copy session c6746058199446968d2496df55cdee42-21a8d5798c8943658113d67c88f4ad25 on tablet 21a8d5798c8943658113d67c88f4ad25 with peer c6746058199446968d2496df55cdee42
I20240502 07:59:04.760804 28479 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.762048 25890 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.763232 25629 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.765431 25354 catalog_manager.cc:5484] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.770777 28850 ts_tablet_manager.cc:902] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:04.772476 28850 tablet_copy_client.cc:308] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:04.775050 28776 log.cc:824] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:04.783125 25518 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet a1fa713b432740d98bcab918d251b54d from peer c5f3003b08fb4214b116c0be8dd62851 ({username='slave'} at 127.18.13.133:46279)
I20240502 07:59:04.783285 25518 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet a1fa713b432740d98bcab918d251b54d from peer c5f3003b08fb4214b116c0be8dd62851 at {username='slave'} at 127.18.13.133:46279: session id = c5f3003b08fb4214b116c0be8dd62851-a1fa713b432740d98bcab918d251b54d
I20240502 07:59:04.784324 25518 tablet_copy_source_session.cc:205] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.785591 28850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fa713b432740d98bcab918d251b54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.791805 25758 consensus_queue.cc:237] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
W20240502 07:59:04.794045 25695 consensus_peers.cc:485] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03016ecfa11d4987a2e9d2368a66a5a2. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.794358 25629 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 07:59:04.794756 25696 consensus_peers.cc:485] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03016ecfa11d4987a2e9d2368a66a5a2. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.794775 28705 ts_tablet_manager.cc:902] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38: Initiating tablet copy from peer 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:59:04.794876 28464 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.795300 28705 tablet_copy_client.cc:308] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:38825
I20240502 07:59:04.795840 28643 ts_tablet_manager.cc:902] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:04.796296 28643 tablet_copy_client.cc:308] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:04.796475 28850 tablet_copy_client.cc:783] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.796690 28850 tablet_copy_client.cc:647] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.800350 25649 tablet_copy_service.cc:140] P 0dfa00c061c748b18772d5f0566944b5: Received BeginTabletCopySession request for tablet c81a7795712e45b5a7fd97450a9f06a7 from peer 85fbc6c98f0643469c92215a5cfb2e38 ({username='slave'} at 127.18.13.136:49679)
I20240502 07:59:04.800478 25649 tablet_copy_service.cc:161] P 0dfa00c061c748b18772d5f0566944b5: Beginning new tablet copy session on tablet c81a7795712e45b5a7fd97450a9f06a7 from peer 85fbc6c98f0643469c92215a5cfb2e38 at {username='slave'} at 127.18.13.136:49679: session id = 85fbc6c98f0643469c92215a5cfb2e38-c81a7795712e45b5a7fd97450a9f06a7
I20240502 07:59:04.801285 25649 tablet_copy_source_session.cc:205] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.801489 25780 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet 9722dae4e4414084bdd8066b41bcc47a from peer c6746058199446968d2496df55cdee42 ({username='slave'} at 127.18.13.135:52983)
I20240502 07:59:04.801591 25780 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet 9722dae4e4414084bdd8066b41bcc47a from peer c6746058199446968d2496df55cdee42 at {username='slave'} at 127.18.13.135:52983: session id = c6746058199446968d2496df55cdee42-9722dae4e4414084bdd8066b41bcc47a
I20240502 07:59:04.802317 28705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c81a7795712e45b5a7fd97450a9f06a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.802559 25780 tablet_copy_source_session.cc:205] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.803407 28643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9722dae4e4414084bdd8066b41bcc47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.804562 28820 tablet_bootstrap.cc:492] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:04.804612 28705 tablet_copy_client.cc:783] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.804755 28705 tablet_copy_client.cc:647] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.804847 28820 tablet_bootstrap.cc:492] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42: Bootstrap complete.
I20240502 07:59:04.805014 28820 ts_tablet_manager.cc:1367] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42: Time spent bootstrapping tablet: real 0.093s user 0.029s sys 0.000s
I20240502 07:59:04.805589 28643 tablet_copy_client.cc:783] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.805730 28643 tablet_copy_client.cc:647] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.807163 25496 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.808084 28465 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.811247 27589 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.812052 25628 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.814291 25347 catalog_manager.cc:5484] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.814997 25496 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:04.827637 28643 tablet_copy_client.cc:523] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:04.829176 28643 tablet_bootstrap.cc:492] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42: Bootstrap starting.
I20240502 07:59:04.805480 28820 raft_consensus.cc:348] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:04.830034 28820 raft_consensus.cc:729] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6746058199446968d2496df55cdee42, State: Initialized, Role: LEARNER
I20240502 07:59:04.830262 28820 consensus_queue.cc:260] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:04.831568 28820 ts_tablet_manager.cc:1398] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240502 07:59:04.833567 25518 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session c6746058199446968d2496df55cdee42-056ba8d15d18409e946c905b8876097a received from {username='slave'} at 127.18.13.135:42665
I20240502 07:59:04.833676 25518 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session c6746058199446968d2496df55cdee42-056ba8d15d18409e946c905b8876097a on tablet 056ba8d15d18409e946c905b8876097a with peer c6746058199446968d2496df55cdee42
I20240502 07:59:04.835562 28820 tablet_replica.cc:330] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: stopping tablet replica
I20240502 07:59:04.841217 27407 raft_consensus.cc:1203] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:04.842065 28850 tablet_copy_client.cc:523] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:04.851768 28850 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851: Bootstrap starting.
I20240502 07:59:04.853081 28820 raft_consensus.cc:2229] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 07:59:04.854136 28820 pending_rounds.cc:62] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Trying to abort 1 pending ops.
I20240502 07:59:04.854311 28820 pending_rounds.cc:68] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023152104903188480 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "a1fa713b432740d98bcab918d251b54d" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } } }
I20240502 07:59:04.854890 28820 raft_consensus.cc:2894] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 07:59:04.855965 28705 tablet_copy_client.cc:523] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:04.856133 28820 raft_consensus.cc:2866] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 07:59:04.856266 28820 raft_consensus.cc:2258] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 07:59:04.857605 28705 tablet_bootstrap.cc:492] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap starting.
I20240502 07:59:04.859004 28820 ts_tablet_manager.cc:1875] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:04.861235 28895 ts_tablet_manager.cc:902] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:04.861884 28895 tablet_copy_client.cc:308] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:04.862560 25518 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet 8904fa47fc274c14acd291b4c91964be from peer 85fbc6c98f0643469c92215a5cfb2e38 ({username='slave'} at 127.18.13.136:58689)
I20240502 07:59:04.862673 25518 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet 8904fa47fc274c14acd291b4c91964be from peer 85fbc6c98f0643469c92215a5cfb2e38 at {username='slave'} at 127.18.13.136:58689: session id = 85fbc6c98f0643469c92215a5cfb2e38-8904fa47fc274c14acd291b4c91964be
I20240502 07:59:04.863956 25518 tablet_copy_source_session.cc:205] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.867990 25628 consensus_queue.cc:237] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:04.870086 25496 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.870846 28828 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 07:59:04.871017 25564 consensus_peers.cc:485] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ad22d2e464d4defab6bd0061e6058f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.871981 25758 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.872407 28586 ts_tablet_manager.cc:902] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:04.872926 28586 tablet_copy_client.cc:308] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:04.873391 28820 ts_tablet_manager.cc:1888] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:04.873486 28820 log.cc:1196] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal/wals/a1fa713b432740d98bcab918d251b54d
I20240502 07:59:04.873816 28820 ts_tablet_manager.cc:902] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
W20240502 07:59:04.874172 25563 consensus_peers.cc:485] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ad22d2e464d4defab6bd0061e6058f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.874270 28820 tablet_copy_client.cc:308] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:04.878217 28828 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.880340 28828 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.885146 25496 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.886701 25780 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet 9722dae4e4414084bdd8066b41bcc47a from peer a17bbff6a71b4667b14f5dd12bca3057 ({username='slave'} at 127.18.13.134:33979)
I20240502 07:59:04.886822 25780 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet 9722dae4e4414084bdd8066b41bcc47a from peer a17bbff6a71b4667b14f5dd12bca3057 at {username='slave'} at 127.18.13.134:33979: session id = a17bbff6a71b4667b14f5dd12bca3057-9722dae4e4414084bdd8066b41bcc47a
I20240502 07:59:04.887866 25780 tablet_copy_source_session.cc:205] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.888994 25518 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet a1fa713b432740d98bcab918d251b54d from peer c6746058199446968d2496df55cdee42 ({username='slave'} at 127.18.13.135:42665)
I20240502 07:59:04.889119 25518 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet a1fa713b432740d98bcab918d251b54d from peer c6746058199446968d2496df55cdee42 at {username='slave'} at 127.18.13.135:42665: session id = c6746058199446968d2496df55cdee42-a1fa713b432740d98bcab918d251b54d
I20240502 07:59:04.889389 28586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9722dae4e4414084bdd8066b41bcc47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.890043 25518 tablet_copy_source_session.cc:205] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.892483 25348 catalog_manager.cc:5484] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 4 to 5, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.893127 28895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8904fa47fc274c14acd291b4c91964be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.895877 25758 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.898550 28924 ts_tablet_manager.cc:902] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:04.899451 28924 tablet_copy_client.cc:308] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:04.900523 25518 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet ee25634bf534443b90fdbdcebdbcb02e from peer 85fbc6c98f0643469c92215a5cfb2e38 ({username='slave'} at 127.18.13.136:58689)
I20240502 07:59:04.903453 25518 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet ee25634bf534443b90fdbdcebdbcb02e from peer 85fbc6c98f0643469c92215a5cfb2e38 at {username='slave'} at 127.18.13.136:58689: session id = 85fbc6c98f0643469c92215a5cfb2e38-ee25634bf534443b90fdbdcebdbcb02e
I20240502 07:59:04.904187 28820 ts_tablet_manager.cc:1875] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 07:59:04.904606 25518 tablet_copy_source_session.cc:205] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.891760 28586 tablet_copy_client.cc:783] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.905968 28586 tablet_copy_client.cc:647] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.906774 28820 ts_tablet_manager.cc:1888] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 07:59:04.907049 28820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fa713b432740d98bcab918d251b54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.907666 28924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee25634bf534443b90fdbdcebdbcb02e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.914342 28705 tablet_bootstrap.cc:492] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:04.914669 28705 tablet_bootstrap.cc:492] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap complete.
I20240502 07:59:04.914849 28705 ts_tablet_manager.cc:1367] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent bootstrapping tablet: real 0.057s user 0.026s sys 0.000s
I20240502 07:59:04.915661 28895 tablet_copy_client.cc:783] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.916416 28895 tablet_copy_client.cc:647] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.918988 28895 tablet_copy_client.cc:523] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:04.920679 28895 tablet_bootstrap.cc:492] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap starting.
I20240502 07:59:04.925180 28820 tablet_copy_client.cc:783] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.925383 28820 tablet_copy_client.cc:647] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.925787 28586 tablet_copy_client.cc:523] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:04.927420 28586 tablet_bootstrap.cc:492] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap starting.
I20240502 07:59:04.928416 28820 tablet_copy_client.cc:523] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:04.928982 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.929953 28820 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Bootstrap starting.
I20240502 07:59:04.931458 28705 raft_consensus.cc:348] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:04.931632 28705 raft_consensus.cc:729] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85fbc6c98f0643469c92215a5cfb2e38, State: Initialized, Role: LEARNER
I20240502 07:59:04.931764 28705 consensus_queue.cc:260] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:04.932858 28705 ts_tablet_manager.cc:1398] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20240502 07:59:04.933357 25649 tablet_copy_service.cc:342] P 0dfa00c061c748b18772d5f0566944b5: Request end of tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-c81a7795712e45b5a7fd97450a9f06a7 received from {username='slave'} at 127.18.13.136:49679
I20240502 07:59:04.933435 25649 tablet_copy_service.cc:433] P 0dfa00c061c748b18772d5f0566944b5: ending tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-c81a7795712e45b5a7fd97450a9f06a7 on tablet c81a7795712e45b5a7fd97450a9f06a7 with peer 85fbc6c98f0643469c92215a5cfb2e38
I20240502 07:59:04.937749 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:04.949339 28885 ts_tablet_manager.cc:902] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:04.949944 28885 tablet_copy_client.cc:308] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:04.950608 25780 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0 from peer c6746058199446968d2496df55cdee42 ({username='slave'} at 127.18.13.135:52983)
I20240502 07:59:04.950726 25780 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0 from peer c6746058199446968d2496df55cdee42 at {username='slave'} at 127.18.13.135:52983: session id = c6746058199446968d2496df55cdee42-e5958bb5c8fc4c1d91ae8f1e8b3861d0
I20240502 07:59:04.952674 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:04.951737 25780 tablet_copy_source_session.cc:205] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.957206 28885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.959626 28885 tablet_copy_client.cc:783] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.960255 28885 tablet_copy_client.cc:647] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.961649 28797 tablet_bootstrap.cc:492] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:04.961999 25758 consensus_queue.cc:237] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:04.963114 25760 consensus_queue.cc:237] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:04.963976 25890 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.964403 28797 tablet_bootstrap.cc:492] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap complete.
I20240502 07:59:04.964973 25629 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.965411 25628 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.965533 28880 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.965791 28797 ts_tablet_manager.cc:1367] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38: Time spent bootstrapping tablet: real 0.214s user 0.029s sys 0.003s
I20240502 07:59:04.966320 28797 raft_consensus.cc:348] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
W20240502 07:59:04.966758 25696 consensus_peers.cc:485] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 273e203d4ed44c43859f08e9bec08a9d. This is attempt 1: this message will repeat every 5th retry.
W20240502 07:59:04.966885 25696 consensus_peers.cc:485] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3bc416015dc40d88743215d2cf46230. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.967105 28464 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.967393 28464 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 07:59:04.967686 25694 consensus_peers.cc:485] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3bc416015dc40d88743215d2cf46230. This is attempt 1: this message will repeat every 5th retry.
W20240502 07:59:04.967823 25694 consensus_peers.cc:485] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 273e203d4ed44c43859f08e9bec08a9d. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:04.968593 28797 raft_consensus.cc:729] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85fbc6c98f0643469c92215a5cfb2e38, State: Initialized, Role: LEARNER
I20240502 07:59:04.968605 25890 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:04.969058 28797 consensus_queue.cc:260] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:04.969372 28880 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:04.969682 28880 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.970362 28797 ts_tablet_manager.cc:1398] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 07:59:04.971002 28586 tablet_bootstrap.cc:492] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:04.971347 28586 tablet_bootstrap.cc:492] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap complete.
I20240502 07:59:04.971549 28586 ts_tablet_manager.cc:1367] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057: Time spent bootstrapping tablet: real 0.044s user 0.027s sys 0.002s
I20240502 07:59:04.972096 28586 raft_consensus.cc:348] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:04.972241 28586 raft_consensus.cc:729] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a17bbff6a71b4667b14f5dd12bca3057, State: Initialized, Role: LEARNER
I20240502 07:59:04.972357 28586 consensus_queue.cc:260] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:04.973193 28643 tablet_bootstrap.cc:492] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:04.973398 25348 catalog_manager.cc:5484] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.974270 25890 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.976125 28586 ts_tablet_manager.cc:1398] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 07:59:04.976794 25780 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session a17bbff6a71b4667b14f5dd12bca3057-9722dae4e4414084bdd8066b41bcc47a received from {username='slave'} at 127.18.13.134:33979
I20240502 07:59:04.976963 25780 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session a17bbff6a71b4667b14f5dd12bca3057-9722dae4e4414084bdd8066b41bcc47a on tablet 9722dae4e4414084bdd8066b41bcc47a with peer a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:04.977095 28776 tablet_bootstrap.cc:492] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:04.977469 28643 tablet_bootstrap.cc:492] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42: Bootstrap complete.
I20240502 07:59:04.977702 28643 ts_tablet_manager.cc:1367] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42: Time spent bootstrapping tablet: real 0.149s user 0.029s sys 0.001s
I20240502 07:59:04.978250 28643 raft_consensus.cc:348] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:04.978415 28643 raft_consensus.cc:729] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6746058199446968d2496df55cdee42, State: Initialized, Role: LEARNER
I20240502 07:59:04.978534 28643 consensus_queue.cc:260] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:04.978612 25518 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-056ba8d15d18409e946c905b8876097a received from {username='slave'} at 127.18.13.136:58689
I20240502 07:59:04.978703 25518 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-056ba8d15d18409e946c905b8876097a on tablet 056ba8d15d18409e946c905b8876097a with peer 85fbc6c98f0643469c92215a5cfb2e38
I20240502 07:59:04.979507 28643 ts_tablet_manager.cc:1398] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:04.980104 28776 tablet_bootstrap.cc:492] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851: Bootstrap complete.
I20240502 07:59:04.980654 28776 ts_tablet_manager.cc:1367] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851: Time spent bootstrapping tablet: real 0.226s user 0.036s sys 0.009s
I20240502 07:59:04.983655 25780 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session c6746058199446968d2496df55cdee42-9722dae4e4414084bdd8066b41bcc47a received from {username='slave'} at 127.18.13.135:52983
I20240502 07:59:04.983767 25780 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session c6746058199446968d2496df55cdee42-9722dae4e4414084bdd8066b41bcc47a on tablet 9722dae4e4414084bdd8066b41bcc47a with peer c6746058199446968d2496df55cdee42
I20240502 07:59:04.984269 25629 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.984673 28880 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.985643 28776 raft_consensus.cc:348] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:04.986011 28776 raft_consensus.cc:729] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5f3003b08fb4214b116c0be8dd62851, State: Initialized, Role: LEARNER
I20240502 07:59:04.986466 28776 consensus_queue.cc:260] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:04.987551 28776 ts_tablet_manager.cc:1398] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 07:59:04.988433 25518 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session c5f3003b08fb4214b116c0be8dd62851-eb0c33c3e4014eb3b931d64adac60e1a received from {username='slave'} at 127.18.13.133:46279
I20240502 07:59:04.988521 25518 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session c5f3003b08fb4214b116c0be8dd62851-eb0c33c3e4014eb3b931d64adac60e1a on tablet eb0c33c3e4014eb3b931d64adac60e1a with peer c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:04.988761 25890 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:04.989717 28776 ts_tablet_manager.cc:902] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:04.989990 25350 catalog_manager.cc:5484] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 4 to 5, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:04.990164 28776 tablet_copy_client.cc:308] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:04.990764 25518 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet 5eb2de2942914800bdbe924d45a9b1e7 from peer c5f3003b08fb4214b116c0be8dd62851 ({username='slave'} at 127.18.13.133:46279)
I20240502 07:59:04.990875 25518 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet 5eb2de2942914800bdbe924d45a9b1e7 from peer c5f3003b08fb4214b116c0be8dd62851 at {username='slave'} at 127.18.13.133:46279: session id = c5f3003b08fb4214b116c0be8dd62851-5eb2de2942914800bdbe924d45a9b1e7
I20240502 07:59:04.991838 25518 tablet_copy_source_session.cc:205] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:04.992903 28776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5eb2de2942914800bdbe924d45a9b1e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.995090 28820 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:04.994966 25628 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:04.995376 28820 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Bootstrap complete.
I20240502 07:59:04.996515 28820 ts_tablet_manager.cc:1367] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Time spent bootstrapping tablet: real 0.067s user 0.029s sys 0.000s
I20240502 07:59:04.996639 28776 tablet_copy_client.cc:783] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:04.996832 28776 tablet_copy_client.cc:647] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:04.997483 28820 raft_consensus.cc:348] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:04.997637 28820 raft_consensus.cc:729] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6746058199446968d2496df55cdee42, State: Initialized, Role: LEARNER
I20240502 07:59:04.997763 28820 consensus_queue.cc:260] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:04.998157 28820 ts_tablet_manager.cc:1398] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:04.998673 25518 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session c6746058199446968d2496df55cdee42-a1fa713b432740d98bcab918d251b54d received from {username='slave'} at 127.18.13.135:42665
I20240502 07:59:04.998767 25518 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session c6746058199446968d2496df55cdee42-a1fa713b432740d98bcab918d251b54d on tablet a1fa713b432740d98bcab918d251b54d with peer c6746058199446968d2496df55cdee42
I20240502 07:59:04.999897 28776 tablet_copy_client.cc:523] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.001390 28776 tablet_bootstrap.cc:492] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap starting.
I20240502 07:59:05.008493 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:05.024777 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:05.025326 25496 consensus_queue.cc:237] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:05.027276 25760 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:59:05.027452 25431 consensus_peers.cc:485] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8e50f3d37d842868900a4cff5bf3601. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:05.027724 28675 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:05.028028 25890 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:05.028628 28374 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:05.031463 27671 raft_consensus.cc:1203] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:05.032183 27670 raft_consensus.cc:1203] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:05.035935 28776 tablet_bootstrap.cc:492] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.036264 28776 tablet_bootstrap.cc:492] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap complete.
I20240502 07:59:05.036458 28776 ts_tablet_manager.cc:1367] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.000s
I20240502 07:59:05.038717 28675 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:05.039758 25890 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:05.041329 25760 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:05.038386 28776 raft_consensus.cc:348] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:05.042516 28776 raft_consensus.cc:729] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5f3003b08fb4214b116c0be8dd62851, State: Initialized, Role: LEARNER
I20240502 07:59:05.044142 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:05.045852 25347 catalog_manager.cc:5484] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:05.046247 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:05.056268 28977 ts_tablet_manager.cc:902] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:59:05.057544 28820 ts_tablet_manager.cc:902] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:05.057718 28977 tablet_copy_client.cc:308] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.132:45865
I20240502 07:59:05.057991 28820 tablet_copy_client.cc:308] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:05.058277 25911 tablet_copy_service.cc:140] P e9596bfb8d324b4da93a4288771ed177: Received BeginTabletCopySession request for tablet 0aa60608089c4f50bcee339d01815a65 from peer a17bbff6a71b4667b14f5dd12bca3057 ({username='slave'} at 127.18.13.134:49677)
I20240502 07:59:05.058387 25911 tablet_copy_service.cc:161] P e9596bfb8d324b4da93a4288771ed177: Beginning new tablet copy session on tablet 0aa60608089c4f50bcee339d01815a65 from peer a17bbff6a71b4667b14f5dd12bca3057 at {username='slave'} at 127.18.13.134:49677: session id = a17bbff6a71b4667b14f5dd12bca3057-0aa60608089c4f50bcee339d01815a65
I20240502 07:59:05.058506 25517 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet ee25634bf534443b90fdbdcebdbcb02e from peer c6746058199446968d2496df55cdee42 ({username='slave'} at 127.18.13.135:42665)
I20240502 07:59:05.059661 25911 tablet_copy_source_session.cc:205] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.059902 25517 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet ee25634bf534443b90fdbdcebdbcb02e from peer c6746058199446968d2496df55cdee42 at {username='slave'} at 127.18.13.135:42665: session id = c6746058199446968d2496df55cdee42-ee25634bf534443b90fdbdcebdbcb02e
I20240502 07:59:05.060926 25517 tablet_copy_source_session.cc:205] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.062038 28820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee25634bf534443b90fdbdcebdbcb02e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.061959 28776 consensus_queue.cc:260] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:05.062476 28776 ts_tablet_manager.cc:1398] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240502 07:59:05.063822 28885 tablet_copy_client.cc:523] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.065100 28885 tablet_bootstrap.cc:492] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42: Bootstrap starting.
I20240502 07:59:05.065272 25517 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session c5f3003b08fb4214b116c0be8dd62851-5eb2de2942914800bdbe924d45a9b1e7 received from {username='slave'} at 127.18.13.133:46279
I20240502 07:59:05.065346 25517 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session c5f3003b08fb4214b116c0be8dd62851-5eb2de2942914800bdbe924d45a9b1e7 on tablet 5eb2de2942914800bdbe924d45a9b1e7 with peer c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:05.065930 28820 tablet_copy_client.cc:783] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.066143 28820 tablet_copy_client.cc:647] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.067684 28820 tablet_copy_client.cc:523] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.069321 28820 tablet_bootstrap.cc:492] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42: Bootstrap starting.
I20240502 07:59:05.071085 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:05.074486 28776 ts_tablet_manager.cc:902] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:05.078665 28776 tablet_copy_client.cc:308] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:05.079412 25517 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet fd68090fdadf45faa0e3b2936b30047e from peer c5f3003b08fb4214b116c0be8dd62851 ({username='slave'} at 127.18.13.133:46279)
I20240502 07:59:05.079540 25517 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet fd68090fdadf45faa0e3b2936b30047e from peer c5f3003b08fb4214b116c0be8dd62851 at {username='slave'} at 127.18.13.133:46279: session id = c5f3003b08fb4214b116c0be8dd62851-fd68090fdadf45faa0e3b2936b30047e
I20240502 07:59:05.080591 28942 ts_tablet_manager.cc:902] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:05.080665 25517 tablet_copy_source_session.cc:205] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.081288 28942 tablet_copy_client.cc:308] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:05.081735 28776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd68090fdadf45faa0e3b2936b30047e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.083968 28776 tablet_copy_client.cc:783] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.084138 28776 tablet_copy_client.cc:647] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.085558 28776 tablet_copy_client.cc:523] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.086138 28977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa60608089c4f50bcee339d01815a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.086879 28776 tablet_bootstrap.cc:492] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851: Bootstrap starting.
I20240502 07:59:05.089936 25780 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet 87d6266008784883891ca32c01363dbe from peer c5f3003b08fb4214b116c0be8dd62851 ({username='slave'} at 127.18.13.133:39885)
I20240502 07:59:05.090073 25780 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet 87d6266008784883891ca32c01363dbe from peer c5f3003b08fb4214b116c0be8dd62851 at {username='slave'} at 127.18.13.133:39885: session id = c5f3003b08fb4214b116c0be8dd62851-87d6266008784883891ca32c01363dbe
I20240502 07:59:05.090375 28977 tablet_copy_client.cc:783] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.090658 28977 tablet_copy_client.cc:647] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.091161 25780 tablet_copy_source_session.cc:205] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.092442 28942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87d6266008784883891ca32c01363dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.093134 28586 ts_tablet_manager.cc:902] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:05.093673 28586 tablet_copy_client.cc:308] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:05.094625 27539 raft_consensus.cc:1203] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:05.094725 28924 tablet_copy_client.cc:783] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.095270 28924 tablet_copy_client.cc:647] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.095444 28977 tablet_copy_client.cc:523] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.096591 25780 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet 87d6266008784883891ca32c01363dbe from peer a17bbff6a71b4667b14f5dd12bca3057 ({username='slave'} at 127.18.13.134:33979)
I20240502 07:59:05.096715 25780 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet 87d6266008784883891ca32c01363dbe from peer a17bbff6a71b4667b14f5dd12bca3057 at {username='slave'} at 127.18.13.134:33979: session id = a17bbff6a71b4667b14f5dd12bca3057-87d6266008784883891ca32c01363dbe
I20240502 07:59:05.097370 28924 tablet_copy_client.cc:523] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.097642 25780 tablet_copy_source_session.cc:205] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.098834 28586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87d6266008784883891ca32c01363dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.098937 28924 tablet_bootstrap.cc:492] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap starting.
I20240502 07:59:05.099117 27407 raft_consensus.cc:1203] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:05.104014 28586 tablet_copy_client.cc:783] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.104305 28586 tablet_copy_client.cc:647] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.106559 28977 tablet_bootstrap.cc:492] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap starting.
I20240502 07:59:05.108000 28942 tablet_copy_client.cc:783] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.108247 28942 tablet_copy_client.cc:647] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.113056 28586 tablet_copy_client.cc:523] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.114745 28586 tablet_bootstrap.cc:492] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap starting.
I20240502 07:59:05.125106 28942 tablet_copy_client.cc:523] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.126791 28942 tablet_bootstrap.cc:492] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851: Bootstrap starting.
I20240502 07:59:05.128309 28850 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.128612 28850 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851: Bootstrap complete.
I20240502 07:59:05.128793 28850 ts_tablet_manager.cc:1367] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851: Time spent bootstrapping tablet: real 0.277s user 0.024s sys 0.004s
I20240502 07:59:05.129274 28850 raft_consensus.cc:348] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:05.129510 28850 raft_consensus.cc:729] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5f3003b08fb4214b116c0be8dd62851, State: Initialized, Role: LEARNER
I20240502 07:59:05.129673 28850 consensus_queue.cc:260] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:05.130573 28850 ts_tablet_manager.cc:1398] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:05.131743 25517 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session c5f3003b08fb4214b116c0be8dd62851-a1fa713b432740d98bcab918d251b54d received from {username='slave'} at 127.18.13.133:46279
I20240502 07:59:05.131867 25517 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session c5f3003b08fb4214b116c0be8dd62851-a1fa713b432740d98bcab918d251b54d on tablet a1fa713b432740d98bcab918d251b54d with peer c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:05.137945 25496 consensus_queue.cc:237] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:05.139994 25889 raft_consensus.cc:1261] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.141484 27275 raft_consensus.cc:1261] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 07:59:05.142154 25431 consensus_peers.cc:485] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5eb2de2942914800bdbe924d45a9b1e7. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:05.142364 26968 consensus_queue.cc:1035] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.143558 26968 consensus_queue.cc:1035] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.143862 25760 raft_consensus.cc:1261] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.144308 26968 consensus_queue.cc:1035] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.146191 26968 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:05.148299 25889 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:05.150074 25760 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:05.151046 25347 catalog_manager.cc:5484] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:05.153261 28375 raft_consensus.cc:1050] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER c5f3003b08fb4214b116c0be8dd62851 to VOTER
I20240502 07:59:05.153769 28375 consensus_queue.cc:237] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:05.154196 27275 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
W20240502 07:59:05.158423 25431 consensus_peers.cc:485] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5eb2de2942914800bdbe924d45a9b1e7. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:05.160327 25760 raft_consensus.cc:1261] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.161093 25889 raft_consensus.cc:1261] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.161983 26968 consensus_queue.cc:1035] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.162355 26968 consensus_queue.cc:1035] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.164654 27275 raft_consensus.cc:1261] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.165123 28374 consensus_queue.cc:1035] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.169967 28374 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:05.172626 28850 ts_tablet_manager.cc:902] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: Initiating tablet copy from peer e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:59:05.172816 25890 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:05.173333 28850 tablet_copy_client.cc:308] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.132:45865
I20240502 07:59:05.174410 27276 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:05.176002 25353 catalog_manager.cc:5484] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:05.178680 25911 tablet_copy_service.cc:140] P e9596bfb8d324b4da93a4288771ed177: Received BeginTabletCopySession request for tablet 0aa60608089c4f50bcee339d01815a65 from peer c5f3003b08fb4214b116c0be8dd62851 ({username='slave'} at 127.18.13.133:39133)
I20240502 07:59:05.178790 25911 tablet_copy_service.cc:161] P e9596bfb8d324b4da93a4288771ed177: Beginning new tablet copy session on tablet 0aa60608089c4f50bcee339d01815a65 from peer c5f3003b08fb4214b116c0be8dd62851 at {username='slave'} at 127.18.13.133:39133: session id = c5f3003b08fb4214b116c0be8dd62851-0aa60608089c4f50bcee339d01815a65
I20240502 07:59:05.178889 25760 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:05.179787 25911 tablet_copy_source_session.cc:205] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.182452 28850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa60608089c4f50bcee339d01815a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.183430 25496 consensus_queue.cc:237] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:05.185622 27275 raft_consensus.cc:1261] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.185822 25890 raft_consensus.cc:1261] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.186089 28374 consensus_queue.cc:1035] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.186254 26968 consensus_queue.cc:1035] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.187350 28850 tablet_copy_client.cc:783] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.187635 28850 tablet_copy_client.cc:647] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.188083 25760 raft_consensus.cc:1261] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.188539 28374 consensus_queue.cc:1035] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.188930 28374 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.189352 25890 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.189965 28776 tablet_bootstrap.cc:492] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.190244 28776 tablet_bootstrap.cc:492] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851: Bootstrap complete.
I20240502 07:59:05.190433 28776 ts_tablet_manager.cc:1367] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851: Time spent bootstrapping tablet: real 0.104s user 0.022s sys 0.005s
I20240502 07:59:05.190989 28776 raft_consensus.cc:348] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:05.191143 28776 raft_consensus.cc:729] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5f3003b08fb4214b116c0be8dd62851, State: Initialized, Role: LEARNER
I20240502 07:59:05.191268 28776 consensus_queue.cc:260] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:05.191591 27275 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.192111 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5eb2de2942914800bdbe924d45a9b1e7 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:05.192231 25760 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.192993 25354 catalog_manager.cc:5484] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 6 to 7, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.194978 28776 ts_tablet_manager.cc:1398] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240502 07:59:05.199842 25517 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session c5f3003b08fb4214b116c0be8dd62851-fd68090fdadf45faa0e3b2936b30047e received from {username='slave'} at 127.18.13.133:46279
I20240502 07:59:05.199945 25517 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session c5f3003b08fb4214b116c0be8dd62851-fd68090fdadf45faa0e3b2936b30047e on tablet fd68090fdadf45faa0e3b2936b30047e with peer c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:05.200412 28977 tablet_bootstrap.cc:492] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.201093 25496 consensus_queue.cc:237] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:05.202710 27275 raft_consensus.cc:1261] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:05.202709 25890 raft_consensus.cc:1261] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:05.203109 28375 consensus_queue.cc:1035] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:05.203483 26968 consensus_queue.cc:1035] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:05.204929 28977 tablet_bootstrap.cc:492] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap complete.
I20240502 07:59:05.205145 28977 ts_tablet_manager.cc:1367] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057: Time spent bootstrapping tablet: real 0.099s user 0.030s sys 0.000s
I20240502 07:59:05.205261 28971 ts_tablet_manager.cc:902] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:05.205220 28375 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.205770 28977 raft_consensus.cc:348] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:05.205899 28977 raft_consensus.cc:729] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a17bbff6a71b4667b14f5dd12bca3057, State: Initialized, Role: LEARNER
I20240502 07:59:05.205780 28971 tablet_copy_client.cc:308] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:05.206012 28977 consensus_queue.cc:260] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:05.206708 25517 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet fd68090fdadf45faa0e3b2936b30047e from peer a17bbff6a71b4667b14f5dd12bca3057 ({username='slave'} at 127.18.13.134:48333)
I20240502 07:59:05.206810 25517 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet fd68090fdadf45faa0e3b2936b30047e from peer a17bbff6a71b4667b14f5dd12bca3057 at {username='slave'} at 127.18.13.134:48333: session id = a17bbff6a71b4667b14f5dd12bca3057-fd68090fdadf45faa0e3b2936b30047e
I20240502 07:59:05.207386 28977 ts_tablet_manager.cc:1398] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:05.207425 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5eb2de2942914800bdbe924d45a9b1e7 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:05.207701 25517 tablet_copy_source_session.cc:205] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.209883 25911 tablet_copy_service.cc:342] P e9596bfb8d324b4da93a4288771ed177: Request end of tablet copy session a17bbff6a71b4667b14f5dd12bca3057-0aa60608089c4f50bcee339d01815a65 received from {username='slave'} at 127.18.13.134:49677
I20240502 07:59:05.210618 28971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd68090fdadf45faa0e3b2936b30047e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.210619 25911 tablet_copy_service.cc:433] P e9596bfb8d324b4da93a4288771ed177: ending tablet copy session a17bbff6a71b4667b14f5dd12bca3057-0aa60608089c4f50bcee339d01815a65 on tablet 0aa60608089c4f50bcee339d01815a65 with peer a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:05.209877 25890 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.213202 28971 tablet_copy_client.cc:783] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.213354 28971 tablet_copy_client.cc:647] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.214548 27275 raft_consensus.cc:2932] T 5eb2de2942914800bdbe924d45a9b1e7 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.215058 27387 tablet_service.cc:1514] Processing DeleteTablet for tablet 5eb2de2942914800bdbe924d45a9b1e7 with delete_type TABLET_DATA_TOMBSTONED (TS a17bbff6a71b4667b14f5dd12bca3057 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:43502
W20240502 07:59:05.216538 25340 catalog_manager.cc:4797] TS a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): delete failed for tablet 5eb2de2942914800bdbe924d45a9b1e7 because the tablet was not found. No further retry: Not found: Tablet not found: 5eb2de2942914800bdbe924d45a9b1e7
I20240502 07:59:05.218016 28850 tablet_copy_client.cc:523] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.216425 25353 catalog_manager.cc:5484] T 5eb2de2942914800bdbe924d45a9b1e7 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 7 to 8, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.223290 25738 tablet_service.cc:1514] Processing DeleteTablet for tablet 5eb2de2942914800bdbe924d45a9b1e7 with delete_type TABLET_DATA_TOMBSTONED (TS 14d8fd1fc3a04d9d967b8033df84f34a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:55602
I20240502 07:59:05.228781 28850 tablet_bootstrap.cc:492] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap starting.
I20240502 07:59:05.238189 29060 tablet_replica.cc:330] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a: stopping tablet replica
I20240502 07:59:05.239061 29060 raft_consensus.cc:2229] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:05.239434 29060 raft_consensus.cc:2258] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:05.244002 29060 ts_tablet_manager.cc:1875] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:05.253435 28586 tablet_bootstrap.cc:492] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.253793 28586 tablet_bootstrap.cc:492] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap complete.
I20240502 07:59:05.254010 28586 ts_tablet_manager.cc:1367] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057: Time spent bootstrapping tablet: real 0.139s user 0.024s sys 0.003s
I20240502 07:59:05.257104 28971 tablet_copy_client.cc:523] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.258671 28971 tablet_bootstrap.cc:492] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap starting.
I20240502 07:59:05.266073 28820 tablet_bootstrap.cc:492] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.266252 28797 ts_tablet_manager.cc:902] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38: Initiating tablet copy from peer 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:59:05.266443 28820 tablet_bootstrap.cc:492] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42: Bootstrap complete.
I20240502 07:59:05.266662 28820 ts_tablet_manager.cc:1367] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42: Time spent bootstrapping tablet: real 0.197s user 0.025s sys 0.002s
I20240502 07:59:05.266793 28797 tablet_copy_client.cc:308] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:38825
I20240502 07:59:05.267302 25649 tablet_copy_service.cc:140] P 0dfa00c061c748b18772d5f0566944b5: Received BeginTabletCopySession request for tablet 21a8d5798c8943658113d67c88f4ad25 from peer 85fbc6c98f0643469c92215a5cfb2e38 ({username='slave'} at 127.18.13.136:49679)
I20240502 07:59:05.267429 25649 tablet_copy_service.cc:161] P 0dfa00c061c748b18772d5f0566944b5: Beginning new tablet copy session on tablet 21a8d5798c8943658113d67c88f4ad25 from peer 85fbc6c98f0643469c92215a5cfb2e38 at {username='slave'} at 127.18.13.136:49679: session id = 85fbc6c98f0643469c92215a5cfb2e38-21a8d5798c8943658113d67c88f4ad25
I20240502 07:59:05.268527 25649 tablet_copy_source_session.cc:205] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.269582 28797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21a8d5798c8943658113d67c88f4ad25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.272285 28797 tablet_copy_client.cc:783] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.246953 29060 ts_tablet_manager.cc:1888] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 07:59:05.273314 29060 log.cc:1196] T 5eb2de2942914800bdbe924d45a9b1e7 P 14d8fd1fc3a04d9d967b8033df84f34a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/5eb2de2942914800bdbe924d45a9b1e7
I20240502 07:59:05.274186 25341 catalog_manager.cc:4830] TS 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205): tablet 5eb2de2942914800bdbe924d45a9b1e7 (table rebalance_test_table_6 [id=67841c5c645f4e86970122328fd1238e]) successfully deleted
I20240502 07:59:05.272874 28797 tablet_copy_client.cc:647] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.267190 28820 raft_consensus.cc:348] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:05.280643 28820 raft_consensus.cc:729] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6746058199446968d2496df55cdee42, State: Initialized, Role: LEARNER
I20240502 07:59:05.280283 27539 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.280882 28820 consensus_queue.cc:260] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:05.281258 28820 ts_tablet_manager.cc:1398] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240502 07:59:05.282972 25517 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session c6746058199446968d2496df55cdee42-ee25634bf534443b90fdbdcebdbcb02e received from {username='slave'} at 127.18.13.135:42665
I20240502 07:59:05.283072 25517 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session c6746058199446968d2496df55cdee42-ee25634bf534443b90fdbdcebdbcb02e on tablet ee25634bf534443b90fdbdcebdbcb02e with peer c6746058199446968d2496df55cdee42
I20240502 07:59:05.290436 28797 tablet_copy_client.cc:523] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.292317 28797 tablet_bootstrap.cc:492] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap starting.
I20240502 07:59:05.301941 28924 tablet_bootstrap.cc:492] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.303750 28924 tablet_bootstrap.cc:492] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap complete.
I20240502 07:59:05.307123 28924 ts_tablet_manager.cc:1367] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38: Time spent bootstrapping tablet: real 0.208s user 0.022s sys 0.009s
I20240502 07:59:05.308879 28924 raft_consensus.cc:348] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:05.310046 28586 raft_consensus.cc:348] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:05.310317 28586 raft_consensus.cc:729] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a17bbff6a71b4667b14f5dd12bca3057, State: Initialized, Role: LEARNER
I20240502 07:59:05.310438 28586 consensus_queue.cc:260] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:05.310760 28586 ts_tablet_manager.cc:1398] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057: Time spent starting tablet: real 0.057s user 0.003s sys 0.000s
I20240502 07:59:05.312173 25780 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session a17bbff6a71b4667b14f5dd12bca3057-87d6266008784883891ca32c01363dbe received from {username='slave'} at 127.18.13.134:33979
I20240502 07:59:05.312284 25780 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session a17bbff6a71b4667b14f5dd12bca3057-87d6266008784883891ca32c01363dbe on tablet 87d6266008784883891ca32c01363dbe with peer a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:05.312889 28924 raft_consensus.cc:729] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85fbc6c98f0643469c92215a5cfb2e38, State: Initialized, Role: LEARNER
I20240502 07:59:05.313159 28924 consensus_queue.cc:260] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:05.315554 28924 ts_tablet_manager.cc:1398] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 07:59:05.319725 25517 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-ee25634bf534443b90fdbdcebdbcb02e received from {username='slave'} at 127.18.13.136:58689
I20240502 07:59:05.319866 25517 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-ee25634bf534443b90fdbdcebdbcb02e on tablet ee25634bf534443b90fdbdcebdbcb02e with peer 85fbc6c98f0643469c92215a5cfb2e38
I20240502 07:59:05.321821 25496 consensus_queue.cc:237] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.324859 25760 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.325615 25890 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.326721 27275 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 07:59:05.327279 25431 consensus_peers.cc:485] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240502 07:59:05.327965 25432 consensus_peers.cc:485] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd68090fdadf45faa0e3b2936b30047e. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:05.328075 26968 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.328842 27539 raft_consensus.cc:1203] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:05.329218 28885 tablet_bootstrap.cc:492] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.329517 28885 tablet_bootstrap.cc:492] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42: Bootstrap complete.
I20240502 07:59:05.329705 28885 ts_tablet_manager.cc:1367] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42: Time spent bootstrapping tablet: real 0.265s user 0.028s sys 0.000s
I20240502 07:59:05.330229 27670 raft_consensus.cc:1203] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:05.330264 28375 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.331194 26968 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.336946 26968 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.338196 28885 raft_consensus.cc:348] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:05.339851 25760 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.341071 28820 ts_tablet_manager.cc:902] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:05.341534 28820 tablet_copy_client.cc:308] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:05.341068 28885 raft_consensus.cc:729] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6746058199446968d2496df55cdee42, State: Initialized, Role: LEARNER
I20240502 07:59:05.341941 28885 consensus_queue.cc:260] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:05.342271 28375 raft_consensus.cc:1050] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER c5f3003b08fb4214b116c0be8dd62851 to VOTER
I20240502 07:59:05.342368 28885 ts_tablet_manager.cc:1398] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 07:59:05.342555 25780 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet 03016ecfa11d4987a2e9d2368a66a5a2 from peer c6746058199446968d2496df55cdee42 ({username='slave'} at 127.18.13.135:52983)
I20240502 07:59:05.342653 25780 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet 03016ecfa11d4987a2e9d2368a66a5a2 from peer c6746058199446968d2496df55cdee42 at {username='slave'} at 127.18.13.135:52983: session id = c6746058199446968d2496df55cdee42-03016ecfa11d4987a2e9d2368a66a5a2
I20240502 07:59:05.342809 25779 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session c6746058199446968d2496df55cdee42-e5958bb5c8fc4c1d91ae8f1e8b3861d0 received from {username='slave'} at 127.18.13.135:52983
I20240502 07:59:05.342746 28375 consensus_queue.cc:237] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.342868 25779 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session c6746058199446968d2496df55cdee42-e5958bb5c8fc4c1d91ae8f1e8b3861d0 on tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0 with peer c6746058199446968d2496df55cdee42
I20240502 07:59:05.343602 25780 tablet_copy_source_session.cc:205] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.344532 28820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03016ecfa11d4987a2e9d2368a66a5a2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 07:59:05.345397 25431 consensus_peers.cc:485] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:05.345536 25354 catalog_manager.cc:5484] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 5 to 6, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.345664 27275 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.346752 28820 tablet_copy_client.cc:783] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.346904 28820 tablet_copy_client.cc:647] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.347154 27276 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.348042 28675 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.349464 28776 ts_tablet_manager.cc:902] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:05.350004 28776 tablet_copy_client.cc:308] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:05.350883 25889 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.350898 25758 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.351505 28675 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.351680 28374 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.352636 25780 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet 03016ecfa11d4987a2e9d2368a66a5a2 from peer c5f3003b08fb4214b116c0be8dd62851 ({username='slave'} at 127.18.13.133:39885)
I20240502 07:59:05.352739 25780 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet 03016ecfa11d4987a2e9d2368a66a5a2 from peer c5f3003b08fb4214b116c0be8dd62851 at {username='slave'} at 127.18.13.133:39885: session id = c5f3003b08fb4214b116c0be8dd62851-03016ecfa11d4987a2e9d2368a66a5a2
I20240502 07:59:05.353799 25780 tablet_copy_source_session.cc:205] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.353832 25889 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.354485 28820 tablet_copy_client.cc:523] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.354802 28776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03016ecfa11d4987a2e9d2368a66a5a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.355886 28820 tablet_bootstrap.cc:492] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42: Bootstrap starting.
I20240502 07:59:05.356724 28374 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.357411 28776 tablet_copy_client.cc:783] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.357556 28776 tablet_copy_client.cc:647] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.358955 27276 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
W20240502 07:59:05.351665 25432 consensus_peers.cc:485] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd68090fdadf45faa0e3b2936b30047e. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:05.360047 28942 tablet_bootstrap.cc:492] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.362447 27539 raft_consensus.cc:1203] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:05.362979 25353 catalog_manager.cc:5484] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851 reported cstate change: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.363704 27276 raft_consensus.cc:1203] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:05.364619 25889 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.367272 25760 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.368108 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:05.368345 28776 tablet_copy_client.cc:523] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.369943 28776 tablet_bootstrap.cc:492] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap starting.
I20240502 07:59:05.375944 25496 consensus_queue.cc:237] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.377794 27276 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:05.378661 25889 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:05.379158 28515 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:05.379343 28374 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 07:59:05.379657 25432 consensus_peers.cc:485] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd68090fdadf45faa0e3b2936b30047e. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:05.382696 28515 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.384852 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fd68090fdadf45faa0e3b2936b30047e with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:05.385710 25889 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.386457 25353 catalog_manager.cc:5484] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 7 to 8, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:05.387948 27276 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.390239 27387 tablet_service.cc:1514] Processing DeleteTablet for tablet fd68090fdadf45faa0e3b2936b30047e with delete_type TABLET_DATA_TOMBSTONED (TS a17bbff6a71b4667b14f5dd12bca3057 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:43502
I20240502 07:59:05.390776 25496 consensus_queue.cc:237] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
W20240502 07:59:05.392342 25340 catalog_manager.cc:4810] TS a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): delete failed for tablet fd68090fdadf45faa0e3b2936b30047e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd68090fdadf45faa0e3b2936b30047e already in progress: copying tablet
I20240502 07:59:05.392657 25890 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:05.392917 25628 consensus_queue.cc:237] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:05.393312 25758 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:05.393466 28675 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:05.393882 28675 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:05.394366 25890 raft_consensus.cc:1261] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:05.394766 28942 tablet_bootstrap.cc:492] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851: Bootstrap complete.
I20240502 07:59:05.394781 25758 raft_consensus.cc:1261] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:05.394789 28825 consensus_queue.cc:1035] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:05.395054 28942 ts_tablet_manager.cc:1367] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851: Time spent bootstrapping tablet: real 0.268s user 0.029s sys 0.004s
I20240502 07:59:05.395149 28461 consensus_queue.cc:1035] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:05.395272 25759 raft_consensus.cc:858] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
W20240502 07:59:05.395325 25759 raft_consensus.cc:1565] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:05.395591 28942 raft_consensus.cc:348] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:05.395728 28942 raft_consensus.cc:729] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5f3003b08fb4214b116c0be8dd62851, State: Initialized, Role: LEARNER
I20240502 07:59:05.395648 25759 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.395857 28942 consensus_queue.cc:260] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:05.396443 28942 ts_tablet_manager.cc:1398] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:05.397770 28924 ts_tablet_manager.cc:902] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38: Initiating tablet copy from peer 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:59:05.398247 28924 tablet_copy_client.cc:308] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:38825
I20240502 07:59:05.398377 28971 tablet_bootstrap.cc:492] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.398692 28971 tablet_bootstrap.cc:492] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap complete.
I20240502 07:59:05.398942 28971 ts_tablet_manager.cc:1367] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057: Time spent bootstrapping tablet: real 0.140s user 0.025s sys 0.005s
I20240502 07:59:05.399219 27276 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:05.399398 28971 raft_consensus.cc:348] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:05.399523 28971 raft_consensus.cc:729] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a17bbff6a71b4667b14f5dd12bca3057, State: Initialized, Role: LEARNER
I20240502 07:59:05.399627 28971 consensus_queue.cc:260] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:05.399684 28374 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:05.399772 28895 tablet_bootstrap.cc:492] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.399977 25649 tablet_copy_service.cc:140] P 0dfa00c061c748b18772d5f0566944b5: Received BeginTabletCopySession request for tablet 1ad22d2e464d4defab6bd0061e6058f8 from peer 85fbc6c98f0643469c92215a5cfb2e38 ({username='slave'} at 127.18.13.136:49679)
I20240502 07:59:05.400064 28895 tablet_bootstrap.cc:492] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap complete.
I20240502 07:59:05.400081 25649 tablet_copy_service.cc:161] P 0dfa00c061c748b18772d5f0566944b5: Beginning new tablet copy session on tablet 1ad22d2e464d4defab6bd0061e6058f8 from peer 85fbc6c98f0643469c92215a5cfb2e38 at {username='slave'} at 127.18.13.136:49679: session id = 85fbc6c98f0643469c92215a5cfb2e38-1ad22d2e464d4defab6bd0061e6058f8
I20240502 07:59:05.400238 28895 ts_tablet_manager.cc:1367] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38: Time spent bootstrapping tablet: real 0.480s user 0.027s sys 0.003s
I20240502 07:59:05.400676 28895 raft_consensus.cc:348] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.400782 28895 raft_consensus.cc:729] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85fbc6c98f0643469c92215a5cfb2e38, State: Initialized, Role: LEARNER
I20240502 07:59:05.400874 28895 consensus_queue.cc:260] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.401078 25649 tablet_copy_source_session.cc:205] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.401113 28971 ts_tablet_manager.cc:1398] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:05.401520 25517 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session a17bbff6a71b4667b14f5dd12bca3057-fd68090fdadf45faa0e3b2936b30047e received from {username='slave'} at 127.18.13.134:48333
I20240502 07:59:05.401593 25517 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session a17bbff6a71b4667b14f5dd12bca3057-fd68090fdadf45faa0e3b2936b30047e on tablet fd68090fdadf45faa0e3b2936b30047e with peer a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:05.402271 28924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad22d2e464d4defab6bd0061e6058f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.402520 26968 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.404268 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fd68090fdadf45faa0e3b2936b30047e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:05.404939 28924 tablet_copy_client.cc:783] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.405223 28924 tablet_copy_client.cc:647] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.406798 25889 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.407584 28461 raft_consensus.cc:2932] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:05.408085 27387 tablet_service.cc:1514] Processing DeleteTablet for tablet fd68090fdadf45faa0e3b2936b30047e with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:43502
I20240502 07:59:05.408318 29059 tablet_replica.cc:330] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057: stopping tablet replica
I20240502 07:59:05.408703 29059 raft_consensus.cc:2229] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 07:59:05.408957 29059 raft_consensus.cc:2258] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 07:59:05.409930 25889 raft_consensus.cc:2932] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:05.410039 25350 catalog_manager.cc:5484] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 8 to 9, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:05.413138 29059 ts_tablet_manager.cc:1875] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:05.413385 27407 raft_consensus.cc:1203] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:05.414381 28924 tablet_copy_client.cc:523] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.415594 29059 ts_tablet_manager.cc:1888] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:05.415874 28924 tablet_bootstrap.cc:492] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap starting.
I20240502 07:59:05.415908 29059 log.cc:1196] T fd68090fdadf45faa0e3b2936b30047e P a17bbff6a71b4667b14f5dd12bca3057: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal/wals/fd68090fdadf45faa0e3b2936b30047e
I20240502 07:59:05.416612 25340 catalog_manager.cc:4830] TS a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): tablet fd68090fdadf45faa0e3b2936b30047e (table rebalance_test_table_3 [id=7c7fb9171c9548a7ad2d45e295d780cb]) successfully deleted
I20240502 07:59:05.417268 25496 consensus_queue.cc:237] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:05.418368 27387 tablet_service.cc:1514] Processing DeleteTablet for tablet fd68090fdadf45faa0e3b2936b30047e with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 9)) from {username='slave'} at 127.0.0.1:43502
I20240502 07:59:05.419751 28885 ts_tablet_manager.cc:902] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: Initiating tablet copy from peer 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:59:05.419780 25340 catalog_manager.cc:4830] TS a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): tablet fd68090fdadf45faa0e3b2936b30047e (table rebalance_test_table_3 [id=7c7fb9171c9548a7ad2d45e295d780cb]) successfully deleted
I20240502 07:59:05.420261 28885 tablet_copy_client.cc:308] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:38825
I20240502 07:59:05.420734 25890 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:05.421115 25649 tablet_copy_service.cc:140] P 0dfa00c061c748b18772d5f0566944b5: Received BeginTabletCopySession request for tablet 1ad22d2e464d4defab6bd0061e6058f8 from peer c6746058199446968d2496df55cdee42 ({username='slave'} at 127.18.13.135:56049)
I20240502 07:59:05.421223 25649 tablet_copy_service.cc:161] P 0dfa00c061c748b18772d5f0566944b5: Beginning new tablet copy session on tablet 1ad22d2e464d4defab6bd0061e6058f8 from peer c6746058199446968d2496df55cdee42 at {username='slave'} at 127.18.13.135:56049: session id = c6746058199446968d2496df55cdee42-1ad22d2e464d4defab6bd0061e6058f8
I20240502 07:59:05.421159 25353 catalog_manager.cc:5484] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:05.422189 25649 tablet_copy_source_session.cc:205] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.423245 28885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad22d2e464d4defab6bd0061e6058f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.423962 25760 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.425165 27275 raft_consensus.cc:1261] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:05.425704 28885 tablet_copy_client.cc:783] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.425637 28374 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:05.425858 28885 tablet_copy_client.cc:647] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.426440 27275 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.427961 28776 tablet_bootstrap.cc:492] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.428247 28776 tablet_bootstrap.cc:492] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap complete.
I20240502 07:59:05.428423 28776 ts_tablet_manager.cc:1367] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851: Time spent bootstrapping tablet: real 0.059s user 0.029s sys 0.000s
I20240502 07:59:05.428913 28776 raft_consensus.cc:348] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:05.429018 28776 raft_consensus.cc:729] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5f3003b08fb4214b116c0be8dd62851, State: Initialized, Role: LEARNER
I20240502 07:59:05.429148 28776 consensus_queue.cc:260] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:05.429502 28776 ts_tablet_manager.cc:1398] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:05.432102 27274 raft_consensus.cc:1203] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:05.434280 28515 consensus_queue.cc:1035] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:05.435961 25779 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session c5f3003b08fb4214b116c0be8dd62851-03016ecfa11d4987a2e9d2368a66a5a2 received from {username='slave'} at 127.18.13.133:39885
I20240502 07:59:05.436053 25779 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session c5f3003b08fb4214b116c0be8dd62851-03016ecfa11d4987a2e9d2368a66a5a2 on tablet 03016ecfa11d4987a2e9d2368a66a5a2 with peer c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:05.436906 28375 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.438670 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fd68090fdadf45faa0e3b2936b30047e with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:05.438999 27275 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.441460 25779 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session c5f3003b08fb4214b116c0be8dd62851-87d6266008784883891ca32c01363dbe received from {username='slave'} at 127.18.13.133:39885
I20240502 07:59:05.441548 25779 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session c5f3003b08fb4214b116c0be8dd62851-87d6266008784883891ca32c01363dbe on tablet 87d6266008784883891ca32c01363dbe with peer c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:05.442374 25348 catalog_manager.cc:5484] T fd68090fdadf45faa0e3b2936b30047e P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 9 to 10, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:05.443878 25758 raft_consensus.cc:2932] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:05.451776 25738 tablet_service.cc:1514] Processing DeleteTablet for tablet fd68090fdadf45faa0e3b2936b30047e with delete_type TABLET_DATA_TOMBSTONED (TS 14d8fd1fc3a04d9d967b8033df84f34a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55602
I20240502 07:59:05.452065 29060 tablet_replica.cc:330] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a: stopping tablet replica
I20240502 07:59:05.452217 29060 raft_consensus.cc:2229] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:05.452447 29060 raft_consensus.cc:2258] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 07:59:05.452580 25565 consensus_peers.cc:485] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc1ac05112724e0ea1f84f648598304e. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:05.452936 25889 raft_consensus.cc:2932] T fd68090fdadf45faa0e3b2936b30047e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.453998 27651 tablet_service.cc:1514] Processing DeleteTablet for tablet fd68090fdadf45faa0e3b2936b30047e with delete_type TABLET_DATA_TOMBSTONED (TS 85fbc6c98f0643469c92215a5cfb2e38 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37636
I20240502 07:59:05.455013 29060 ts_tablet_manager.cc:1875] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 07:59:05.455204 25341 catalog_manager.cc:4797] TS 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): delete failed for tablet fd68090fdadf45faa0e3b2936b30047e because the tablet was not found. No further retry: Not found: Tablet not found: fd68090fdadf45faa0e3b2936b30047e
I20240502 07:59:05.457592 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:05.459371 29060 ts_tablet_manager.cc:1888] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 07:59:05.459443 29060 log.cc:1196] T fd68090fdadf45faa0e3b2936b30047e P 14d8fd1fc3a04d9d967b8033df84f34a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/fd68090fdadf45faa0e3b2936b30047e
I20240502 07:59:05.460281 25341 catalog_manager.cc:4830] TS 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205): tablet fd68090fdadf45faa0e3b2936b30047e (table rebalance_test_table_3 [id=7c7fb9171c9548a7ad2d45e295d780cb]) successfully deleted
I20240502 07:59:05.466240 28850 tablet_bootstrap.cc:492] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.466574 28850 tablet_bootstrap.cc:492] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap complete.
I20240502 07:59:05.466810 28850 ts_tablet_manager.cc:1367] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: Time spent bootstrapping tablet: real 0.238s user 0.025s sys 0.004s
I20240502 07:59:05.467334 28850 raft_consensus.cc:348] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:05.467479 28850 raft_consensus.cc:729] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5f3003b08fb4214b116c0be8dd62851, State: Initialized, Role: LEARNER
I20240502 07:59:05.467592 28850 consensus_queue.cc:260] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:05.469471 28850 ts_tablet_manager.cc:1398] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:05.470028 25911 tablet_copy_service.cc:342] P e9596bfb8d324b4da93a4288771ed177: Request end of tablet copy session c5f3003b08fb4214b116c0be8dd62851-0aa60608089c4f50bcee339d01815a65 received from {username='slave'} at 127.18.13.133:39133
I20240502 07:59:05.470113 25911 tablet_copy_service.cc:433] P e9596bfb8d324b4da93a4288771ed177: ending tablet copy session c5f3003b08fb4214b116c0be8dd62851-0aa60608089c4f50bcee339d01815a65 on tablet 0aa60608089c4f50bcee339d01815a65 with peer c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:05.473120 26712 raft_consensus.cc:1050] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177: attempting to promote NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 to VOTER
I20240502 07:59:05.473680 27407 raft_consensus.cc:1203] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:05.473714 28895 ts_tablet_manager.cc:1398] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38: Time spent starting tablet: real 0.073s user 0.000s sys 0.001s
I20240502 07:59:05.474808 28479 raft_consensus.cc:1050] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a: attempting to promote NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 to VOTER
I20240502 07:59:05.474851 26712 consensus_queue.cc:237] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:05.475270 28479 consensus_queue.cc:237] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:05.475693 25517 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-8904fa47fc274c14acd291b4c91964be received from {username='slave'} at 127.18.13.136:58689
I20240502 07:59:05.475768 25517 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-8904fa47fc274c14acd291b4c91964be on tablet 8904fa47fc274c14acd291b4c91964be with peer 85fbc6c98f0643469c92215a5cfb2e38
I20240502 07:59:05.476549 25496 raft_consensus.cc:1261] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.476763 25889 raft_consensus.cc:1261] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.476972 25628 raft_consensus.cc:1261] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.477169 28464 consensus_queue.cc:1035] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.477260 25629 raft_consensus.cc:1261] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.477428 27589 consensus_queue.cc:1035] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.478073 27407 raft_consensus.cc:1261] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.478410 26712 consensus_queue.cc:1035] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.478741 26712 consensus_queue.cc:1035] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.478992 26712 consensus_queue.cc:1035] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.479888 29161 ts_tablet_manager.cc:902] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:05.484669 29115 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.486452 25496 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.489446 25353 catalog_manager.cc:5484] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 4 to 5, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.489657 27407 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.490173 27539 raft_consensus.cc:1203] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:05.492738 25628 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.497051 25889 consensus_queue.cc:237] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:05.499007 25496 raft_consensus.cc:1261] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.499872 29115 consensus_queue.cc:1035] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.499738 27407 raft_consensus.cc:1261] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.501617 29115 consensus_queue.cc:1035] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.502910 27589 raft_consensus.cc:2932] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.503474 29115 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.504043 27407 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.505746 25342 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d6929d2560404acb9bb53f54a181f958 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:05.506188 25496 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.506556 25758 consensus_queue.cc:237] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.507372 25889 raft_consensus.cc:2932] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.508487 25629 raft_consensus.cc:1261] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:05.509058 28960 consensus_queue.cc:1035] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:59:05.509439 25694 consensus_peers.cc:485] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99c3ddedadbf4c0e8858085b66022342. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:05.509137 25348 catalog_manager.cc:5484] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 5 to 6, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:05.510712 25628 raft_consensus.cc:2932] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.511274 25347 catalog_manager.cc:5484] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:05.512416 29161 tablet_copy_client.cc:308] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:05.512814 29109 raft_consensus.cc:1050] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 to VOTER
I20240502 07:59:05.512892 25779 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet 273e203d4ed44c43859f08e9bec08a9d from peer 85fbc6c98f0643469c92215a5cfb2e38 ({username='slave'} at 127.18.13.136:56619)
I20240502 07:59:05.512985 25779 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet 273e203d4ed44c43859f08e9bec08a9d from peer 85fbc6c98f0643469c92215a5cfb2e38 at {username='slave'} at 127.18.13.136:56619: session id = 85fbc6c98f0643469c92215a5cfb2e38-273e203d4ed44c43859f08e9bec08a9d
I20240502 07:59:05.513298 29109 consensus_queue.cc:237] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:05.514035 25779 tablet_copy_source_session.cc:205] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.515235 25890 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.515275 27539 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.515544 27670 raft_consensus.cc:1261] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.515825 25496 raft_consensus.cc:1261] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:05.516832 28375 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.517192 25607 tablet_service.cc:1514] Processing DeleteTablet for tablet d6929d2560404acb9bb53f54a181f958 with delete_type TABLET_DATA_TOMBSTONED (TS 0dfa00c061c748b18772d5f0566944b5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34854
I20240502 07:59:05.517225 28375 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.518816 28880 consensus_queue.cc:1035] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.518931 29198 tablet_replica.cc:330] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5: stopping tablet replica
I20240502 07:59:05.519068 27670 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.519253 29198 raft_consensus.cc:2229] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:05.519512 29198 raft_consensus.cc:2258] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:05.519572 28515 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.520051 28880 consensus_queue.cc:1035] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:05.520921 29198 ts_tablet_manager.cc:1875] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:05.524052 29198 ts_tablet_manager.cc:1888] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:05.524226 29198 log.cc:1196] T d6929d2560404acb9bb53f54a181f958 P 0dfa00c061c748b18772d5f0566944b5: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/d6929d2560404acb9bb53f54a181f958
I20240502 07:59:05.525291 27670 raft_consensus.cc:2932] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.525805 25343 catalog_manager.cc:4830] TS 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): tablet d6929d2560404acb9bb53f54a181f958 (table rebalance_test_table_5 [id=a5520c28ad46406f9fdf6855d5dfa37b]) successfully deleted
I20240502 07:59:05.528612 25628 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.529103 28515 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.531018 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:05.535060 28885 tablet_copy_client.cc:523] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.536613 28885 tablet_bootstrap.cc:492] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: Bootstrap starting.
I20240502 07:59:05.540910 28850 ts_tablet_manager.cc:902] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:05.541361 28850 tablet_copy_client.cc:308] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:05.541901 25779 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet 273e203d4ed44c43859f08e9bec08a9d from peer c5f3003b08fb4214b116c0be8dd62851 ({username='slave'} at 127.18.13.133:39885)
I20240502 07:59:05.542023 25779 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet 273e203d4ed44c43859f08e9bec08a9d from peer c5f3003b08fb4214b116c0be8dd62851 at {username='slave'} at 127.18.13.133:39885: session id = c5f3003b08fb4214b116c0be8dd62851-273e203d4ed44c43859f08e9bec08a9d
I20240502 07:59:05.543021 25779 tablet_copy_source_session.cc:205] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.543970 28850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273e203d4ed44c43859f08e9bec08a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.544031 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:05.546343 28850 tablet_copy_client.cc:783] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.546531 28850 tablet_copy_client.cc:647] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.548769 29161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273e203d4ed44c43859f08e9bec08a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.551237 29161 tablet_copy_client.cc:783] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.551476 29161 tablet_copy_client.cc:647] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.552556 28515 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:05.555650 28850 tablet_copy_client.cc:523] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.556617 27671 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:05.557178 28850 tablet_bootstrap.cc:492] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851: Bootstrap starting.
I20240502 07:59:05.559036 28895 ts_tablet_manager.cc:902] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:05.559058 25890 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:05.559865 28895 tablet_copy_client.cc:308] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:05.561103 25348 catalog_manager.cc:5484] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38 reported cstate change: config changed from index 5 to 6, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:05.564177 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:05.564303 28942 ts_tablet_manager.cc:902] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:05.564883 28942 tablet_copy_client.cc:308] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:05.571493 25780 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet a3bc416015dc40d88743215d2cf46230 from peer c5f3003b08fb4214b116c0be8dd62851 ({username='slave'} at 127.18.13.133:39885)
I20240502 07:59:05.571646 25780 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet a3bc416015dc40d88743215d2cf46230 from peer c5f3003b08fb4214b116c0be8dd62851 at {username='slave'} at 127.18.13.133:39885: session id = c5f3003b08fb4214b116c0be8dd62851-a3bc416015dc40d88743215d2cf46230
I20240502 07:59:05.572664 25780 tablet_copy_source_session.cc:205] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.573312 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:05.574398 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:05.575627 28942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bc416015dc40d88743215d2cf46230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.578361 28942 tablet_copy_client.cc:783] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.578588 28942 tablet_copy_client.cc:647] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.579620 27276 raft_consensus.cc:1203] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:05.583664 27538 raft_consensus.cc:1203] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:05.586064 26968 raft_consensus.cc:1050] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER c6746058199446968d2496df55cdee42 to VOTER
I20240502 07:59:05.586556 26968 consensus_queue.cc:237] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:05.588315 27671 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.588670 27539 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:05.588801 29108 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.590595 27538 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.591219 25890 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.592195 28375 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.592514 28375 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.594314 25779 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet a3bc416015dc40d88743215d2cf46230 from peer 85fbc6c98f0643469c92215a5cfb2e38 ({username='slave'} at 127.18.13.136:56619)
I20240502 07:59:05.594434 25779 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet a3bc416015dc40d88743215d2cf46230 from peer 85fbc6c98f0643469c92215a5cfb2e38 at {username='slave'} at 127.18.13.136:56619: session id = 85fbc6c98f0643469c92215a5cfb2e38-a3bc416015dc40d88743215d2cf46230
I20240502 07:59:05.595331 25779 tablet_copy_source_session.cc:205] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.597927 28895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3bc416015dc40d88743215d2cf46230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.600592 28895 tablet_copy_client.cc:783] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.600975 28895 tablet_copy_client.cc:647] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.601840 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:05.602298 28515 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:05.603231 28895 tablet_copy_client.cc:523] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.604799 28895 tablet_bootstrap.cc:492] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap starting.
I20240502 07:59:05.605671 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:05.606348 27538 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:05.606407 25890 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:05.608552 27671 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:05.609683 25627 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.610467 29108 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.611233 25627 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:05.611882 25354 catalog_manager.cc:5484] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42 reported cstate change: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:05.613055 25627 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:05.615635 27407 raft_consensus.cc:1203] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:05.618227 29109 raft_consensus.cc:1050] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 to VOTER
I20240502 07:59:05.618680 29109 consensus_queue.cc:237] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:05.620266 25758 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.620261 27407 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.620668 29108 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.620847 28675 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.622643 25627 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.622783 25496 consensus_queue.cc:237] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:05.624049 28675 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.624346 25627 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:05.624477 27538 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:05.624720 27671 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:05.626423 28675 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:05.627103 28675 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:05.629900 28675 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:05.632195 29108 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:05.632822 28971 ts_tablet_manager.cc:902] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:05.632691 25628 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:05.633347 28971 tablet_copy_client.cc:308] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:05.634611 25517 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet d8e50f3d37d842868900a4cff5bf3601 from peer a17bbff6a71b4667b14f5dd12bca3057 ({username='slave'} at 127.18.13.134:48333)
I20240502 07:59:05.634708 25517 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet d8e50f3d37d842868900a4cff5bf3601 from peer a17bbff6a71b4667b14f5dd12bca3057 at {username='slave'} at 127.18.13.134:48333: session id = a17bbff6a71b4667b14f5dd12bca3057-d8e50f3d37d842868900a4cff5bf3601
I20240502 07:59:05.635423 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 056ba8d15d18409e946c905b8876097a with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:05.635533 25517 tablet_copy_source_session.cc:205] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.635949 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:05.636649 27671 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:05.636698 27538 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:05.638753 25350 catalog_manager.cc:5484] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:05.639639 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:05.640028 28971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e50f3d37d842868900a4cff5bf3601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.642252 28971 tablet_copy_client.cc:783] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.642403 28971 tablet_copy_client.cc:647] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.646354 28971 tablet_copy_client.cc:523] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.647887 28971 tablet_bootstrap.cc:492] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap starting.
I20240502 07:59:05.648509 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:05.649149 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:05.656627 27276 raft_consensus.cc:1203] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:05.667073 25496 consensus_queue.cc:237] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:05.668494 27538 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:05.668342 28960 raft_consensus.cc:2932] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.668923 25870 tablet_service.cc:1514] Processing DeleteTablet for tablet 056ba8d15d18409e946c905b8876097a with delete_type TABLET_DATA_TOMBSTONED (TS e9596bfb8d324b4da93a4288771ed177 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47880
I20240502 07:59:05.669462 26968 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:05.671561 27671 raft_consensus.cc:1261] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:05.672369 29109 consensus_queue.cc:1035] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:05.672762 29109 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:05.672456 25350 catalog_manager.cc:5484] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:05.674361 25496 raft_consensus.cc:2932] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.674574 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 056ba8d15d18409e946c905b8876097a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:05.675280 27538 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:05.677203 27671 raft_consensus.cc:2932] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:05.677942 25629 raft_consensus.cc:2932] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.681164 25354 catalog_manager.cc:5484] T 056ba8d15d18409e946c905b8876097a P 85fbc6c98f0643469c92215a5cfb2e38 reported cstate change: config changed from index 8 to 9, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:05.684029 25607 tablet_service.cc:1514] Processing DeleteTablet for tablet 056ba8d15d18409e946c905b8876097a with delete_type TABLET_DATA_TOMBSTONED (TS 0dfa00c061c748b18772d5f0566944b5 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34854
I20240502 07:59:05.684302 29198 tablet_replica.cc:330] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5: stopping tablet replica
I20240502 07:59:05.684444 29198 raft_consensus.cc:2229] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:05.684612 29198 raft_consensus.cc:2258] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:05.692560 29198 ts_tablet_manager.cc:1875] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:05.695073 29198 ts_tablet_manager.cc:1888] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 07:59:05.695155 29198 log.cc:1196] T 056ba8d15d18409e946c905b8876097a P 0dfa00c061c748b18772d5f0566944b5: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/056ba8d15d18409e946c905b8876097a
I20240502 07:59:05.695940 25343 catalog_manager.cc:4830] TS 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): tablet 056ba8d15d18409e946c905b8876097a (table rebalance_test_table_2 [id=f9eeb74ff004488ba3049c0e8e535bb5]) successfully deleted
I20240502 07:59:05.697094 29161 tablet_copy_client.cc:523] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.698624 29161 tablet_bootstrap.cc:492] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap starting.
I20240502 07:59:05.710891 28515 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.712556 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:05.713274 25758 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.716012 25347 catalog_manager.cc:5484] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 4 to 5, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:05.717658 27407 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.723678 29229 tablet_replica.cc:330] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177: stopping tablet replica
I20240502 07:59:05.724319 29229 raft_consensus.cc:2229] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:05.724545 27519 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:05.724614 27518 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:05.724673 29229 raft_consensus.cc:2258] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:05.726147 25627 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.728965 28942 tablet_copy_client.cc:523] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.730635 28942 tablet_bootstrap.cc:492] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap starting.
I20240502 07:59:05.733217 29229 ts_tablet_manager.cc:1875] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:05.737886 27276 raft_consensus.cc:1203] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:05.742388 28797 tablet_bootstrap.cc:492] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.742796 28797 tablet_bootstrap.cc:492] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap complete.
I20240502 07:59:05.743072 28797 ts_tablet_manager.cc:1367] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent bootstrapping tablet: real 0.451s user 0.030s sys 0.000s
I20240502 07:59:05.743670 28797 raft_consensus.cc:348] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.744182 28797 raft_consensus.cc:729] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85fbc6c98f0643469c92215a5cfb2e38, State: Initialized, Role: LEARNER
I20240502 07:59:05.744369 28797 consensus_queue.cc:260] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.746002 28797 ts_tablet_manager.cc:1398] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:59:05.746505 25649 tablet_copy_service.cc:342] P 0dfa00c061c748b18772d5f0566944b5: Request end of tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-21a8d5798c8943658113d67c88f4ad25 received from {username='slave'} at 127.18.13.136:49679
I20240502 07:59:05.746591 25649 tablet_copy_service.cc:433] P 0dfa00c061c748b18772d5f0566944b5: ending tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-21a8d5798c8943658113d67c88f4ad25 on tablet 21a8d5798c8943658113d67c88f4ad25 with peer 85fbc6c98f0643469c92215a5cfb2e38
I20240502 07:59:05.756558 28375 raft_consensus.cc:1050] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER c6746058199446968d2496df55cdee42 to VOTER
I20240502 07:59:05.757109 28375 consensus_queue.cc:237] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:05.759552 25758 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.759999 27538 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.760005 25890 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.760964 27276 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.761507 28374 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.761871 28374 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.763360 28675 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.763615 28675 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.774617 28461 raft_consensus.cc:1050] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5: attempting to promote NON_VOTER c6746058199446968d2496df55cdee42 to VOTER
I20240502 07:59:05.774778 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:05.775405 28461 consensus_queue.cc:237] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.777688 27671 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.778242 28825 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.780113 25496 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.780717 27181 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.781462 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:05.782416 29109 raft_consensus.cc:1013] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: attempt to promote peer c5f3003b08fb4214b116c0be8dd62851: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 07:59:05.782977 27539 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.783651 25890 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.784224 28825 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.784523 28825 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.786093 29229 ts_tablet_manager.cc:1888] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 07:59:05.786298 29229 log.cc:1196] T 056ba8d15d18409e946c905b8876097a P e9596bfb8d324b4da93a4288771ed177: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/wals/056ba8d15d18409e946c905b8876097a
I20240502 07:59:05.788372 25342 catalog_manager.cc:4830] TS e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865): tablet 056ba8d15d18409e946c905b8876097a (table rebalance_test_table_2 [id=f9eeb74ff004488ba3049c0e8e535bb5]) successfully deleted
I20240502 07:59:05.797556 28820 tablet_bootstrap.cc:492] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.797719 25889 consensus_queue.cc:237] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:05.798054 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 07:59:05.799531 25824 consensus_peers.cc:485] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbfd23aa383f413cbe4a5ccd8146d2b6. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:05.800230 25496 raft_consensus.cc:1261] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:05.800683 29169 consensus_queue.cc:1035] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:05.800808 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:05.808230 25627 raft_consensus.cc:1261] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:05.808873 29169 consensus_queue.cc:1035] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:05.828995 25497 consensus_queue.cc:237] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:05.831112 25760 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.831573 28515 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.832615 25889 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.832999 28515 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.835400 28515 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:05.837927 25760 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:05.841688 25353 catalog_manager.cc:5484] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:05.846681 25889 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
W20240502 07:59:05.851323 28942 log.cc:924] Time spent T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
W20240502 07:59:05.853236 25433 consensus_peers.cc:485] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8e50f3d37d842868900a4cff5bf3601. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:05.854185 28464 raft_consensus.cc:1050] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a: attempting to promote NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 to VOTER
I20240502 07:59:05.854650 28464 consensus_queue.cc:237] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:05.856487 27407 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.857882 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:05.858222 27539 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.860095 27518 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:05.865046 27670 raft_consensus.cc:1203] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:05.865733 28479 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.866080 28479 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.866956 25627 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.867427 25498 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.867683 28479 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.867909 28960 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.870718 27589 raft_consensus.cc:1013] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: attempt to promote peer c6746058199446968d2496df55cdee42: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240502 07:59:05.871256 25431 consensus_peers.cc:485] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:05.875321 28820 tablet_bootstrap.cc:492] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42: Bootstrap complete.
I20240502 07:59:05.875526 28820 ts_tablet_manager.cc:1367] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42: Time spent bootstrapping tablet: real 0.520s user 0.029s sys 0.000s
I20240502 07:59:05.877197 28820 raft_consensus.cc:348] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:05.877377 28820 raft_consensus.cc:729] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6746058199446968d2496df55cdee42, State: Initialized, Role: LEARNER
I20240502 07:59:05.877506 28820 consensus_queue.cc:260] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:05.878978 28776 ts_tablet_manager.cc:902] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851: Initiating tablet copy from peer 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:59:05.879469 28776 tablet_copy_client.cc:308] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:38825
I20240502 07:59:05.879724 28461 raft_consensus.cc:1050] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5: attempting to promote NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 to VOTER
I20240502 07:59:05.880141 28461 consensus_queue.cc:237] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:05.881722 27670 raft_consensus.cc:1261] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.882091 28825 consensus_queue.cc:1035] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.884225 25889 raft_consensus.cc:1261] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.884713 27181 consensus_queue.cc:1035] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.885016 25760 raft_consensus.cc:1261] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.885444 27181 consensus_queue.cc:1035] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.890141 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:05.894033 28850 tablet_bootstrap.cc:492] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.894433 28850 tablet_bootstrap.cc:492] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851: Bootstrap complete.
I20240502 07:59:05.895210 28960 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.895638 28820 ts_tablet_manager.cc:1398] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42: Time spent starting tablet: real 0.020s user 0.001s sys 0.000s
I20240502 07:59:05.896337 25779 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session c6746058199446968d2496df55cdee42-03016ecfa11d4987a2e9d2368a66a5a2 received from {username='slave'} at 127.18.13.135:52983
I20240502 07:59:05.896397 25779 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session c6746058199446968d2496df55cdee42-03016ecfa11d4987a2e9d2368a66a5a2 on tablet 03016ecfa11d4987a2e9d2368a66a5a2 with peer c6746058199446968d2496df55cdee42
I20240502 07:59:05.897388 27539 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.897428 25497 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.897643 27407 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.899755 28880 raft_consensus.cc:1050] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a: attempting to promote NON_VOTER c6746058199446968d2496df55cdee42 to VOTER
I20240502 07:59:05.900331 25348 catalog_manager.cc:5484] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057 reported cstate change: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.900440 28880 consensus_queue.cc:237] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:05.903227 28461 raft_consensus.cc:1013] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: attempt to promote peer 85fbc6c98f0643469c92215a5cfb2e38: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 07:59:05.903404 27407 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.904428 25497 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.905864 28924 tablet_bootstrap.cc:492] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.906152 28924 tablet_bootstrap.cc:492] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap complete.
I20240502 07:59:05.906311 28924 ts_tablet_manager.cc:1367] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent bootstrapping tablet: real 0.491s user 0.028s sys 0.000s
I20240502 07:59:05.907016 28924 raft_consensus.cc:348] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.907138 28924 raft_consensus.cc:729] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85fbc6c98f0643469c92215a5cfb2e38, State: Initialized, Role: LEARNER
I20240502 07:59:05.907230 28924 consensus_queue.cc:260] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.908376 27589 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.908833 28924 ts_tablet_manager.cc:1398] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:05.909756 25649 tablet_copy_service.cc:342] P 0dfa00c061c748b18772d5f0566944b5: Request end of tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-1ad22d2e464d4defab6bd0061e6058f8 received from {username='slave'} at 127.18.13.136:49679
I20240502 07:59:05.909842 25649 tablet_copy_service.cc:433] P 0dfa00c061c748b18772d5f0566944b5: ending tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-1ad22d2e464d4defab6bd0061e6058f8 on tablet 1ad22d2e464d4defab6bd0061e6058f8 with peer 85fbc6c98f0643469c92215a5cfb2e38
I20240502 07:59:05.910313 27589 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.910948 28825 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.913038 28880 raft_consensus.cc:1050] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a: attempting to promote NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 to VOTER
I20240502 07:59:05.913126 27671 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.913431 28885 tablet_bootstrap.cc:492] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.913492 28880 consensus_queue.cc:237] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:05.913699 28885 tablet_bootstrap.cc:492] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: Bootstrap complete.
I20240502 07:59:05.913862 28885 ts_tablet_manager.cc:1367] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: Time spent bootstrapping tablet: real 0.377s user 0.027s sys 0.001s
I20240502 07:59:05.913719 27537 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.914332 28885 raft_consensus.cc:348] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.914448 28885 raft_consensus.cc:729] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6746058199446968d2496df55cdee42, State: Initialized, Role: LEARNER
I20240502 07:59:05.914541 28885 consensus_queue.cc:260] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.915290 27181 raft_consensus.cc:1050] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5: attempting to promote NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 to VOTER
I20240502 07:59:05.915323 25628 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.915673 27181 consensus_queue.cc:237] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:05.916451 27589 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.917284 25354 catalog_manager.cc:5484] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38 reported cstate change: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.918061 27407 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.918534 28960 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.919219 27671 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.919600 28461 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.921263 28675 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:05.921468 25888 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.921730 25890 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.921687 25891 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.922379 28825 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.922825 28960 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.925132 25758 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:05.925231 25888 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:05.925452 25354 catalog_manager.cc:5484] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20240502 07:59:05.927366 28464 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.929466 25495 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.929371 27408 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.929600 25498 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:05.930826 28895 tablet_bootstrap.cc:492] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.931113 28895 tablet_bootstrap.cc:492] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap complete.
I20240502 07:59:05.931298 28895 ts_tablet_manager.cc:1367] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent bootstrapping tablet: real 0.327s user 0.028s sys 0.000s
I20240502 07:59:05.931744 28895 raft_consensus.cc:348] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.932200 28895 raft_consensus.cc:729] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85fbc6c98f0643469c92215a5cfb2e38, State: Initialized, Role: LEARNER
I20240502 07:59:05.932349 28895 consensus_queue.cc:260] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.896328 28942 tablet_bootstrap.cc:492] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:05.932731 27538 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:05.932935 28942 tablet_bootstrap.cc:492] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap complete.
I20240502 07:59:05.933125 28942 ts_tablet_manager.cc:1367] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851: Time spent bootstrapping tablet: real 0.203s user 0.028s sys 0.000s
I20240502 07:59:05.933264 28895 ts_tablet_manager.cc:1398] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 07:59:05.933594 28942 raft_consensus.cc:348] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.933730 28942 raft_consensus.cc:729] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5f3003b08fb4214b116c0be8dd62851, State: Initialized, Role: LEARNER
I20240502 07:59:05.933813 28942 consensus_queue.cc:260] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:05.934096 28942 ts_tablet_manager.cc:1398] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:05.934696 25779 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session c5f3003b08fb4214b116c0be8dd62851-a3bc416015dc40d88743215d2cf46230 received from {username='slave'} at 127.18.13.133:39885
I20240502 07:59:05.916420 27276 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.934763 25779 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session c5f3003b08fb4214b116c0be8dd62851-a3bc416015dc40d88743215d2cf46230 on tablet a3bc416015dc40d88743215d2cf46230 with peer c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:05.923518 27275 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:05.936421 28375 raft_consensus.cc:1050] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER c5f3003b08fb4214b116c0be8dd62851 to VOTER
I20240502 07:59:05.936575 25779 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-a3bc416015dc40d88743215d2cf46230 received from {username='slave'} at 127.18.13.136:56619
I20240502 07:59:05.936636 25779 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-a3bc416015dc40d88743215d2cf46230 on tablet a3bc416015dc40d88743215d2cf46230 with peer 85fbc6c98f0643469c92215a5cfb2e38
I20240502 07:59:05.936905 28375 consensus_queue.cc:237] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:05.937225 28850 ts_tablet_manager.cc:1367] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851: Time spent bootstrapping tablet: real 0.380s user 0.030s sys 0.001s
I20240502 07:59:05.937716 28850 raft_consensus.cc:348] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:05.937920 28850 raft_consensus.cc:729] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5f3003b08fb4214b116c0be8dd62851, State: Initialized, Role: LEARNER
I20240502 07:59:05.938086 28850 consensus_queue.cc:260] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:05.938115 25350 catalog_manager.cc:5484] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057 reported cstate change: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.938603 28850 ts_tablet_manager.cc:1398] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:05.938839 25758 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.939117 25779 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session c5f3003b08fb4214b116c0be8dd62851-273e203d4ed44c43859f08e9bec08a9d received from {username='slave'} at 127.18.13.133:39885
I20240502 07:59:05.939183 25779 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session c5f3003b08fb4214b116c0be8dd62851-273e203d4ed44c43859f08e9bec08a9d on tablet 273e203d4ed44c43859f08e9bec08a9d with peer c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:05.939325 28374 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.939392 27275 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.939955 28960 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.940218 25888 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.940508 29281 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.940723 28675 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.940966 28515 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.943763 27670 raft_consensus.cc:1203] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:05.944977 28675 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.946089 25888 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.946947 25887 consensus_queue.cc:237] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:05.947521 27536 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.947595 25758 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.949064 25626 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:05.949795 27276 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.950134 29169 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:05.950697 29281 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.929204 25497 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.952044 28825 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.952989 25758 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:05.952893 27671 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
W20240502 07:59:05.953377 25827 consensus_peers.cc:485] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 843d1c1d6384476983749a78f65a5c74. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:05.953913 28479 raft_consensus.cc:1013] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: attempt to promote peer c5f3003b08fb4214b116c0be8dd62851: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 07:59:05.954406 25498 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.954986 25890 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.957247 29169 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:05.957372 25352 catalog_manager.cc:5484] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38 reported cstate change: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.959589 25649 tablet_copy_service.cc:140] P 0dfa00c061c748b18772d5f0566944b5: Received BeginTabletCopySession request for tablet fc1ac05112724e0ea1f84f648598304e from peer c5f3003b08fb4214b116c0be8dd62851 ({username='slave'} at 127.18.13.133:54317)
I20240502 07:59:05.959704 25649 tablet_copy_service.cc:161] P 0dfa00c061c748b18772d5f0566944b5: Beginning new tablet copy session on tablet fc1ac05112724e0ea1f84f648598304e from peer c5f3003b08fb4214b116c0be8dd62851 at {username='slave'} at 127.18.13.133:54317: session id = c5f3003b08fb4214b116c0be8dd62851-fc1ac05112724e0ea1f84f648598304e
I20240502 07:59:05.960762 25649 tablet_copy_source_session.cc:205] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:05.960670 27536 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.961505 28374 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.961748 28776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1ac05112724e0ea1f84f648598304e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:05.959203 27539 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:05.963955 28880 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.958424 28885 ts_tablet_manager.cc:1398] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20240502 07:59:05.964605 25355 catalog_manager.cc:5484] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.965449 25649 tablet_copy_service.cc:342] P 0dfa00c061c748b18772d5f0566944b5: Request end of tablet copy session c6746058199446968d2496df55cdee42-1ad22d2e464d4defab6bd0061e6058f8 received from {username='slave'} at 127.18.13.135:56049
I20240502 07:59:05.965535 25649 tablet_copy_service.cc:433] P 0dfa00c061c748b18772d5f0566944b5: ending tablet copy session c6746058199446968d2496df55cdee42-1ad22d2e464d4defab6bd0061e6058f8 on tablet 1ad22d2e464d4defab6bd0061e6058f8 with peer c6746058199446968d2496df55cdee42
I20240502 07:59:05.962735 27536 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:05.966161 27539 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.966321 28776 tablet_copy_client.cc:783] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:05.966478 28776 tablet_copy_client.cc:647] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:05.968008 28776 tablet_copy_client.cc:523] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:05.968156 25625 consensus_queue.cc:237] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:05.969430 28776 tablet_bootstrap.cc:492] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851: Bootstrap starting.
I20240502 07:59:05.970639 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:05.971045 27671 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:05.971305 25497 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:05.971679 29256 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:05.972620 27536 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:05.972576 25759 consensus_queue.cc:237] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:05.972636 25757 consensus_queue.cc:237] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:05.973001 28825 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:05.973412 29256 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:05.974627 27408 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:05.974839 27538 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.977322 28464 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:05.977579 28825 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.979903 25498 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:05.980300 28960 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:05.981103 25497 consensus_queue.cc:237] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:05.981817 25495 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.982443 27275 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:05.983021 25888 raft_consensus.cc:1261] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.983363 28880 raft_consensus.cc:1050] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a: attempting to promote NON_VOTER c6746058199446968d2496df55cdee42 to VOTER
I20240502 07:59:05.983214 29169 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:05.983480 28374 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:05.983757 28880 consensus_queue.cc:237] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:05.984337 28464 consensus_queue.cc:1035] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.985129 27536 raft_consensus.cc:1261] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.985219 25628 raft_consensus.cc:1261] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.985435 27538 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.985905 25887 raft_consensus.cc:1261] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:05.986701 27671 raft_consensus.cc:1261] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:05.987179 27589 consensus_queue.cc:1035] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.987501 28479 consensus_queue.cc:1035] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:05.987547 27589 consensus_queue.cc:1035] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.987761 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21a8d5798c8943658113d67c88f4ad25 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:05.988018 28479 consensus_queue.cc:1035] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:05.988255 27539 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:05.988327 28479 raft_consensus.cc:2932] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.988332 25352 catalog_manager.cc:5484] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:05.988708 25887 raft_consensus.cc:2932] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.989586 27671 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.992341 28479 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:05.992709 25349 catalog_manager.cc:5484] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:05.994046 28479 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.996672 25341 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71a67cb0f6b54bfb9eb44400fe21ce40 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:05.996861 25341 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9722dae4e4414084bdd8066b41bcc47a with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:05.996953 27275 raft_consensus.cc:1203] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:05.997606 25870 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a8d5798c8943658113d67c88f4ad25 with delete_type TABLET_DATA_TOMBSTONED (TS e9596bfb8d324b4da93a4288771ed177 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47880
I20240502 07:59:05.997820 29229 tablet_replica.cc:330] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177: stopping tablet replica
I20240502 07:59:05.997696 27538 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.997951 29229 raft_consensus.cc:2229] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:05.998123 29229 raft_consensus.cc:2258] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:05.998275 25495 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:05.999327 29229 ts_tablet_manager.cc:1875] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:05.999285 25355 catalog_manager.cc:5484] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 5 to 6, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:05.999662 25625 consensus_queue.cc:237] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:06.001559 29229 ts_tablet_manager.cc:1888] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 07:59:06.001631 29229 log.cc:1196] T 21a8d5798c8943658113d67c88f4ad25 P e9596bfb8d324b4da93a4288771ed177: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/wals/21a8d5798c8943658113d67c88f4ad25
I20240502 07:59:06.001763 27671 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.002166 29256 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.002408 27275 raft_consensus.cc:1203] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:06.002518 25758 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:06.002707 27536 raft_consensus.cc:1261] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.003552 25607 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a67cb0f6b54bfb9eb44400fe21ce40 with delete_type TABLET_DATA_TOMBSTONED (TS 0dfa00c061c748b18772d5f0566944b5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34854
I20240502 07:59:06.004101 29198 tablet_replica.cc:330] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: stopping tablet replica
I20240502 07:59:06.004307 29198 raft_consensus.cc:2229] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.004470 29198 raft_consensus.cc:2258] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.004616 27536 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.005055 28515 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.005168 29198 ts_tablet_manager.cc:1875] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.004072 28825 consensus_queue.cc:1035] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.007328 29198 ts_tablet_manager.cc:1888] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:06.006959 28515 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.007256 29256 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.007961 25758 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.008388 28675 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.009505 25758 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.009761 25355 catalog_manager.cc:5484] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:06.009754 27670 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.011132 27408 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.007398 29198 log.cc:1196] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/71a67cb0f6b54bfb9eb44400fe21ce40
I20240502 07:59:06.013243 25342 catalog_manager.cc:4830] TS e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865): tablet 21a8d5798c8943658113d67c88f4ad25 (table rebalance_test_table_9 [id=10514d1b971c4d5c9e2b1f433f40d2c1]) successfully deleted
I20240502 07:59:06.013702 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a1fa713b432740d98bcab918d251b54d with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.013695 25607 tablet_service.cc:1514] Processing DeleteTablet for tablet 9722dae4e4414084bdd8066b41bcc47a with delete_type TABLET_DATA_TOMBSTONED (TS 0dfa00c061c748b18772d5f0566944b5 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34854
I20240502 07:59:06.013861 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21a8d5798c8943658113d67c88f4ad25 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.013968 25343 catalog_manager.cc:4830] TS 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): tablet 71a67cb0f6b54bfb9eb44400fe21ce40 (table rebalance_test_table_1 [id=a5f087acc4d94b0f9694fb1cf95e7dca]) successfully deleted
I20240502 07:59:06.014093 25355 catalog_manager.cc:5484] T 21a8d5798c8943658113d67c88f4ad25 P 85fbc6c98f0643469c92215a5cfb2e38 reported cstate change: config changed from index 8 to 9, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.014321 25759 consensus_queue.cc:237] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:06.014647 27536 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.015215 29198 tablet_replica.cc:330] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: stopping tablet replica
I20240502 07:59:06.015359 29198 raft_consensus.cc:2229] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.015494 29198 pending_rounds.cc:62] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: Trying to abort 1 pending ops.
I20240502 07:59:06.015560 29198 pending_rounds.cc:68] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023152111019323392 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "9722dae4e4414084bdd8066b41bcc47a" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } } }
I20240502 07:59:06.015729 29198 raft_consensus.cc:2894] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240502 07:59:06.015826 29198 raft_consensus.cc:2866] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 07:59:06.015877 29198 raft_consensus.cc:2258] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.016083 25495 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.016789 28960 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.017359 25477 tablet_service.cc:1514] Processing DeleteTablet for tablet 21a8d5798c8943658113d67c88f4ad25 with delete_type TABLET_DATA_TOMBSTONED (TS c3be11f0fe304e0f8e008dc1e4c41f70 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:50936
I20240502 07:59:06.018479 27536 raft_consensus.cc:2932] T 21a8d5798c8943658113d67c88f4ad25 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.019133 28960 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.019747 25353 catalog_manager.cc:5484] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 reported cstate change: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.022074 25341 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9722dae4e4414084bdd8066b41bcc47a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.022244 27275 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.022610 25495 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.022725 27536 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.022795 25870 tablet_service.cc:1514] Processing DeleteTablet for tablet a1fa713b432740d98bcab918d251b54d with delete_type TABLET_DATA_TOMBSTONED (TS e9596bfb8d324b4da93a4288771ed177 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47880
I20240502 07:59:06.023020 29229 tablet_replica.cc:330] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177: stopping tablet replica
I20240502 07:59:06.023084 28880 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.023160 29229 raft_consensus.cc:2229] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.023335 29229 raft_consensus.cc:2258] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.024286 29229 ts_tablet_manager.cc:1875] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.024318 27536 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.024541 28924 ts_tablet_manager.cc:902] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:06.025084 28924 tablet_copy_client.cc:308] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:06.025774 25779 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet 99c3ddedadbf4c0e8858085b66022342 from peer 85fbc6c98f0643469c92215a5cfb2e38 ({username='slave'} at 127.18.13.136:56619)
I20240502 07:59:06.025873 25779 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet 99c3ddedadbf4c0e8858085b66022342 from peer 85fbc6c98f0643469c92215a5cfb2e38 at {username='slave'} at 127.18.13.136:56619: session id = 85fbc6c98f0643469c92215a5cfb2e38-99c3ddedadbf4c0e8858085b66022342
I20240502 07:59:06.026811 25779 tablet_copy_source_session.cc:205] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:06.026705 27671 raft_consensus.cc:2932] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.027477 25349 catalog_manager.cc:5484] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 8 to 9, VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:06.028414 29229 ts_tablet_manager.cc:1888] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 07:59:06.028736 29229 log.cc:1196] T a1fa713b432740d98bcab918d251b54d P e9596bfb8d324b4da93a4288771ed177: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/wals/a1fa713b432740d98bcab918d251b54d
I20240502 07:59:06.029953 28924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c3ddedadbf4c0e8858085b66022342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:06.032228 28924 tablet_copy_client.cc:783] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:06.032397 28924 tablet_copy_client.cc:647] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:06.033474 27387 tablet_service.cc:1514] Processing DeleteTablet for tablet 9722dae4e4414084bdd8066b41bcc47a with delete_type TABLET_DATA_TOMBSTONED (TS a17bbff6a71b4667b14f5dd12bca3057 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43502
I20240502 07:59:06.033576 28880 raft_consensus.cc:2932] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.034071 27539 raft_consensus.cc:2932] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.035933 25628 raft_consensus.cc:2932] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.039228 25352 catalog_manager.cc:5484] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 4 to 5, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:06.042541 25759 consensus_queue.cc:237] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:06.043671 27539 raft_consensus.cc:1261] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.043677 25628 raft_consensus.cc:1261] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.044059 28479 consensus_queue.cc:1035] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.044306 28464 consensus_queue.cc:1035] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.038246 25890 raft_consensus.cc:2932] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.048034 25342 catalog_manager.cc:4830] TS e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865): tablet a1fa713b432740d98bcab918d251b54d (table rebalance_test_table_2 [id=f9eeb74ff004488ba3049c0e8e535bb5]) successfully deleted
I20240502 07:59:06.049655 25495 consensus_queue.cc:237] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:06.049646 28464 raft_consensus.cc:2932] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.049697 25626 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:06.051175 25341 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.051370 27275 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.051616 25626 raft_consensus.cc:2932] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.051946 29306 tablet_replica.cc:330] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057: stopping tablet replica
I20240502 07:59:06.052084 29306 raft_consensus.cc:2229] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.052613 29306 raft_consensus.cc:2258] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.052933 28374 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.055581 25352 catalog_manager.cc:5484] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 5 to 6, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:06.055755 29306 ts_tablet_manager.cc:1875] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.055779 28374 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.057436 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a1fa713b432740d98bcab918d251b54d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.058676 29300 tablet_replica.cc:330] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70: stopping tablet replica
I20240502 07:59:06.059315 28924 tablet_copy_client.cc:523] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:06.059338 25870 tablet_service.cc:1514] Processing DeleteTablet for tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0 with delete_type TABLET_DATA_TOMBSTONED (TS e9596bfb8d324b4da93a4288771ed177 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47880
I20240502 07:59:06.059083 27275 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.060045 25352 catalog_manager.cc:5484] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 8 to 9, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:06.061756 29306 ts_tablet_manager.cc:1888] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 07:59:06.061900 29306 log.cc:1196] T 9722dae4e4414084bdd8066b41bcc47a P a17bbff6a71b4667b14f5dd12bca3057: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal/wals/9722dae4e4414084bdd8066b41bcc47a
I20240502 07:59:06.062494 29229 tablet_replica.cc:330] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177: stopping tablet replica
I20240502 07:59:06.062541 25340 catalog_manager.cc:4830] TS a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): tablet 9722dae4e4414084bdd8066b41bcc47a (table rebalance_test_table_1 [id=a5f087acc4d94b0f9694fb1cf95e7dca]) successfully deleted
I20240502 07:59:06.062484 27536 raft_consensus.cc:2932] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.062819 29229 raft_consensus.cc:2229] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.063041 29229 raft_consensus.cc:2258] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.063520 28924 tablet_bootstrap.cc:492] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap starting.
I20240502 07:59:06.064007 25738 tablet_service.cc:1514] Processing DeleteTablet for tablet a1fa713b432740d98bcab918d251b54d with delete_type TABLET_DATA_TOMBSTONED (TS 14d8fd1fc3a04d9d967b8033df84f34a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55602
I20240502 07:59:06.064528 29312 tablet_replica.cc:330] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a: stopping tablet replica
I20240502 07:59:06.064734 29312 raft_consensus.cc:2229] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.064771 27536 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.064785 29229 ts_tablet_manager.cc:1875] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.065358 29312 raft_consensus.cc:2258] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.065238 28515 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.065798 29300 raft_consensus.cc:2229] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.066092 29300 raft_consensus.cc:2258] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.066546 27536 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.067301 29229 ts_tablet_manager.cc:1888] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:06.067378 29229 log.cc:1196] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P e9596bfb8d324b4da93a4288771ed177: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/wals/e5958bb5c8fc4c1d91ae8f1e8b3861d0
I20240502 07:59:06.069088 29312 ts_tablet_manager.cc:1875] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.071722 29312 ts_tablet_manager.cc:1888] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 07:59:06.071805 29300 ts_tablet_manager.cc:1875] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.071931 29312 log.cc:1196] T a1fa713b432740d98bcab918d251b54d P 14d8fd1fc3a04d9d967b8033df84f34a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/a1fa713b432740d98bcab918d251b54d
I20240502 07:59:06.072613 25341 catalog_manager.cc:4830] TS 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205): tablet a1fa713b432740d98bcab918d251b54d (table rebalance_test_table_2 [id=f9eeb74ff004488ba3049c0e8e535bb5]) successfully deleted
I20240502 07:59:06.079352 29177 raft_consensus.cc:2932] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:06.082484 25626 raft_consensus.cc:2932] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:06.083587 29300 ts_tablet_manager.cc:1888] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 07:59:06.083513 25496 raft_consensus.cc:2932] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:06.084403 29300 log.cc:1196] T 21a8d5798c8943658113d67c88f4ad25 P c3be11f0fe304e0f8e008dc1e4c41f70: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/21a8d5798c8943658113d67c88f4ad25
I20240502 07:59:06.085362 25347 catalog_manager.cc:5484] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:06.085721 25343 catalog_manager.cc:4830] TS c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377): tablet 21a8d5798c8943658113d67c88f4ad25 (table rebalance_test_table_9 [id=10514d1b971c4d5c9e2b1f433f40d2c1]) successfully deleted
I20240502 07:59:06.098964 29161 tablet_bootstrap.cc:492] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:06.099473 29161 tablet_bootstrap.cc:492] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap complete.
I20240502 07:59:06.101094 28880 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.101639 25891 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.102134 29256 raft_consensus.cc:2932] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.105033 25626 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.105389 27407 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.106602 27275 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.108228 25758 raft_consensus.cc:2932] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.108242 25355 catalog_manager.cc:5484] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057 reported cstate change: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.112365 29161 ts_tablet_manager.cc:1367] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38: Time spent bootstrapping tablet: real 0.414s user 0.032s sys 0.000s
I20240502 07:59:06.113212 29161 raft_consensus.cc:348] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
W20240502 07:59:06.113337 29265 consensus_queue.cc:1159] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0dfa00c061c748b18772d5f0566944b5" status { last_received { term: 1 index: 6 } last_committed_idx: 5 last_received_current_leader { term: 1 index: 6 } }
I20240502 07:59:06.113552 29161 raft_consensus.cc:729] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85fbc6c98f0643469c92215a5cfb2e38, State: Initialized, Role: LEARNER
I20240502 07:59:06.113813 29161 consensus_queue.cc:260] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:06.114574 25342 catalog_manager.cc:4830] TS e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865): tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0 (table rebalance_test_table_3 [id=7c7fb9171c9548a7ad2d45e295d780cb]) successfully deleted
I20240502 07:59:06.114797 29161 ts_tablet_manager.cc:1398] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:06.115482 29198 ts_tablet_manager.cc:1875] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.116358 25349 catalog_manager.cc:5484] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:06.117985 28971 tablet_bootstrap.cc:492] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:06.118320 28971 tablet_bootstrap.cc:492] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap complete.
I20240502 07:59:06.118593 28971 ts_tablet_manager.cc:1367] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057: Time spent bootstrapping tablet: real 0.471s user 0.028s sys 0.000s
I20240502 07:59:06.118635 28880 raft_consensus.cc:1050] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a: attempting to promote NON_VOTER c5f3003b08fb4214b116c0be8dd62851 to VOTER
I20240502 07:59:06.119067 28880 consensus_queue.cc:237] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:06.119115 28971 raft_consensus.cc:348] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:06.119272 28971 raft_consensus.cc:729] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a17bbff6a71b4667b14f5dd12bca3057, State: Initialized, Role: LEARNER
I20240502 07:59:06.119379 28971 consensus_queue.cc:260] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:06.119755 28971 ts_tablet_manager.cc:1398] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:06.120563 25517 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session a17bbff6a71b4667b14f5dd12bca3057-d8e50f3d37d842868900a4cff5bf3601 received from {username='slave'} at 127.18.13.134:48333
I20240502 07:59:06.120671 25517 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session a17bbff6a71b4667b14f5dd12bca3057-d8e50f3d37d842868900a4cff5bf3601 on tablet d8e50f3d37d842868900a4cff5bf3601 with peer a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:06.120983 25629 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.121069 27589 raft_consensus.cc:1050] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a: attempting to promote NON_VOTER c5f3003b08fb4214b116c0be8dd62851 to VOTER
I20240502 07:59:06.121577 27275 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.121490 27589 consensus_queue.cc:237] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:06.121578 27671 raft_consensus.cc:2932] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.123243 28960 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.123601 28479 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.123845 25891 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.124549 27536 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.125042 28479 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.125406 25629 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 07:59:06.126206 25341 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 87d6266008784883891ca32c01363dbe with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:06.126550 27275 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.126703 25629 consensus_queue.cc:237] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:06.127182 29198 ts_tablet_manager.cc:1888] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 07:59:06.127261 29198 log.cc:1196] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/9722dae4e4414084bdd8066b41bcc47a
I20240502 07:59:06.127969 27671 raft_consensus.cc:1261] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.128500 28960 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.128523 29255 consensus_queue.cc:1035] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.128621 28479 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.128962 28960 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.129321 27407 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.130030 25758 raft_consensus.cc:1261] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.130009 28479 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.132040 25496 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.132958 28880 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.134945 25779 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-273e203d4ed44c43859f08e9bec08a9d received from {username='slave'} at 127.18.13.136:56619
I20240502 07:59:06.135017 25779 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-273e203d4ed44c43859f08e9bec08a9d on tablet 273e203d4ed44c43859f08e9bec08a9d with peer 85fbc6c98f0643469c92215a5cfb2e38
I20240502 07:59:06.136299 25889 raft_consensus.cc:2932] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.137852 28825 consensus_queue.cc:1035] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.138518 28825 raft_consensus.cc:2932] T c81a7795712e45b5a7fd97450a9f06a7 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.138661 28479 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.143061 25348 catalog_manager.cc:5484] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:06.144610 25343 catalog_manager.cc:4830] TS 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): tablet 9722dae4e4414084bdd8066b41bcc47a (table rebalance_test_table_1 [id=a5f087acc4d94b0f9694fb1cf95e7dca]) successfully deleted
I20240502 07:59:06.147130 28675 raft_consensus.cc:1050] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER c6746058199446968d2496df55cdee42 to VOTER
I20240502 07:59:06.147344 27274 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.147615 28675 consensus_queue.cc:237] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:06.149399 25627 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.149400 27671 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.149816 27538 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.150121 28375 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.150329 29108 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.151510 27671 raft_consensus.cc:2932] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.151631 29108 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.152129 25889 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.152558 29108 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.154685 25352 catalog_manager.cc:5484] T c81a7795712e45b5a7fd97450a9f06a7 P 85fbc6c98f0643469c92215a5cfb2e38 reported cstate change: config changed from index 5 to 6, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.155856 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c81a7795712e45b5a7fd97450a9f06a7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.156766 25758 raft_consensus.cc:2932] T c81a7795712e45b5a7fd97450a9f06a7 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.157330 25629 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.161388 25891 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.162146 25870 tablet_service.cc:1514] Processing DeleteTablet for tablet c81a7795712e45b5a7fd97450a9f06a7 with delete_type TABLET_DATA_TOMBSTONED (TS e9596bfb8d324b4da93a4288771ed177 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47880
I20240502 07:59:06.162365 29229 tablet_replica.cc:330] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177: stopping tablet replica
I20240502 07:59:06.162493 29229 raft_consensus.cc:2229] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.162657 29229 raft_consensus.cc:2258] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.163440 29229 ts_tablet_manager.cc:1875] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.165582 25758 consensus_queue.cc:237] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:06.165877 29229 ts_tablet_manager.cc:1888] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:06.165944 29229 log.cc:1196] T c81a7795712e45b5a7fd97450a9f06a7 P e9596bfb8d324b4da93a4288771ed177: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/wals/c81a7795712e45b5a7fd97450a9f06a7
I20240502 07:59:06.166500 25342 catalog_manager.cc:4830] TS e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865): tablet c81a7795712e45b5a7fd97450a9f06a7 (table rebalance_test_table_9 [id=10514d1b971c4d5c9e2b1f433f40d2c1]) successfully deleted
I20240502 07:59:06.167354 27274 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.167757 28960 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.169329 25888 consensus_queue.cc:237] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:06.170164 25889 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.170634 28924 tablet_bootstrap.cc:492] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:06.171288 27406 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.171449 27274 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.171877 29168 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.172052 28924 tablet_bootstrap.cc:492] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap complete.
I20240502 07:59:06.172165 25495 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.172184 29168 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.172274 28924 ts_tablet_manager.cc:1367] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent bootstrapping tablet: real 0.109s user 0.029s sys 0.000s
I20240502 07:59:06.172775 28924 raft_consensus.cc:348] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:06.172884 28924 raft_consensus.cc:729] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85fbc6c98f0643469c92215a5cfb2e38, State: Initialized, Role: LEARNER
I20240502 07:59:06.172832 28960 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.172983 28924 consensus_queue.cc:260] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:06.173561 29169 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.175002 25758 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.175259 28924 ts_tablet_manager.cc:1398] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:59:06.175392 29168 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.175523 26968 raft_consensus.cc:1050] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER c5f3003b08fb4214b116c0be8dd62851 to VOTER
I20240502 07:59:06.175999 26968 consensus_queue.cc:237] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:06.177412 25779 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-99c3ddedadbf4c0e8858085b66022342 received from {username='slave'} at 127.18.13.136:56619
I20240502 07:59:06.177488 25779 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-99c3ddedadbf4c0e8858085b66022342 on tablet 99c3ddedadbf4c0e8858085b66022342 with peer 85fbc6c98f0643469c92215a5cfb2e38
I20240502 07:59:06.177660 27275 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:06.177749 25626 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:06.177620 28464 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.178140 29108 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:06.178495 28374 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:06.178613 29108 raft_consensus.cc:1013] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: attempt to promote peer 85fbc6c98f0643469c92215a5cfb2e38: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 07:59:06.178791 25341 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 87d6266008784883891ca32c01363dbe with cas_config_opid_index 6: aborting the task: latest config opid_index 7; task opid_index 6
I20240502 07:59:06.179313 29168 raft_consensus.cc:1013] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: attempt to promote peer a17bbff6a71b4667b14f5dd12bca3057: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 07:59:06.179560 25759 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:06.180218 25341 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 87d6266008784883891ca32c01363dbe with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.180323 27274 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.181017 28375 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:06.181140 25354 catalog_manager.cc:5484] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:06.184567 25607 tablet_service.cc:1514] Processing DeleteTablet for tablet 87d6266008784883891ca32c01363dbe with delete_type TABLET_DATA_TOMBSTONED (TS 0dfa00c061c748b18772d5f0566944b5 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34854
I20240502 07:59:06.185034 29198 tablet_replica.cc:330] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5: stopping tablet replica
I20240502 07:59:06.184939 25888 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.185324 29198 raft_consensus.cc:2229] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.185568 29198 raft_consensus.cc:2258] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.186408 29198 ts_tablet_manager.cc:1875] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.186491 29169 raft_consensus.cc:1013] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: attempt to promote peer c5f3003b08fb4214b116c0be8dd62851: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 07:59:06.187743 28375 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.188465 25626 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.190121 27275 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.190766 25355 catalog_manager.cc:5484] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 4 to 5, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:06.192749 29198 ts_tablet_manager.cc:1888] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 07:59:06.192884 29198 log.cc:1196] T 87d6266008784883891ca32c01363dbe P 0dfa00c061c748b18772d5f0566944b5: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/87d6266008784883891ca32c01363dbe
I20240502 07:59:06.193177 25759 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.193508 25343 catalog_manager.cc:4830] TS 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): tablet 87d6266008784883891ca32c01363dbe (table rebalance_test_table_1 [id=a5f087acc4d94b0f9694fb1cf95e7dca]) successfully deleted
I20240502 07:59:06.194209 27408 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.194849 27589 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.195644 27408 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.196261 25497 consensus_queue.cc:237] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:06.197526 27275 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.197649 25626 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.198004 28375 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.198330 26968 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.199187 27408 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.200512 29109 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.200986 25626 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.201360 27275 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.202452 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb0c33c3e4014eb3b931d64adac60e1a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 07:59:06.203364 25343 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet eb0c33c3e4014eb3b931d64adac60e1a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 5 but the committed config has opid_index of 6
I20240502 07:59:06.203848 25352 catalog_manager.cc:5484] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 reported cstate change: config changed from index 5 to 6, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.206295 25738 tablet_service.cc:1514] Processing DeleteTablet for tablet eb0c33c3e4014eb3b931d64adac60e1a with delete_type TABLET_DATA_TOMBSTONED (TS 14d8fd1fc3a04d9d967b8033df84f34a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55602
I20240502 07:59:06.206562 29312 tablet_replica.cc:330] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a: stopping tablet replica
I20240502 07:59:06.206741 29312 raft_consensus.cc:2229] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.206938 29312 raft_consensus.cc:2258] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.207728 29312 ts_tablet_manager.cc:1875] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.210181 29312 ts_tablet_manager.cc:1888] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:06.210409 28960 raft_consensus.cc:1013] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: attempt to promote peer c6746058199446968d2496df55cdee42: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 07:59:06.210778 28960 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.211184 25496 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.213199 27536 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.213944 25626 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.214666 25348 catalog_manager.cc:5484] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.215540 28880 raft_consensus.cc:1050] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a: attempting to promote NON_VOTER c6746058199446968d2496df55cdee42 to VOTER
I20240502 07:59:06.215955 28880 consensus_queue.cc:237] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:06.217603 27275 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
W20240502 07:59:06.218540 25826 consensus_peers.cc:485] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0aa60608089c4f50bcee339d01815a65. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:06.219372 29108 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.219918 25627 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.221665 27275 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.221779 25495 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.222141 27536 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.222476 25890 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.223429 27589 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.223696 25627 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.224218 25348 catalog_manager.cc:5484] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.225018 27589 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.225358 27589 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.225883 27670 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.226737 27589 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.227831 27536 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.229779 27589 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.230073 29109 raft_consensus.cc:1050] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 to VOTER
I20240502 07:59:06.230554 29109 consensus_queue.cc:237] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:06.232633 25890 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.232825 27670 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.232705 25627 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.233124 28515 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.233386 27539 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.233547 28515 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.233826 25626 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.234248 29108 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.234388 28515 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.235129 27275 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.235513 29312 log.cc:1196] T eb0c33c3e4014eb3b931d64adac60e1a P 14d8fd1fc3a04d9d967b8033df84f34a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/eb0c33c3e4014eb3b931d64adac60e1a
I20240502 07:59:06.236150 25341 catalog_manager.cc:4830] TS 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205): tablet eb0c33c3e4014eb3b931d64adac60e1a (table rebalance_test_table_6 [id=67841c5c645f4e86970122328fd1238e]) successfully deleted
I20240502 07:59:06.236999 29169 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.238059 28515 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.238135 25347 catalog_manager.cc:5484] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.239657 25495 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.241160 27538 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.242611 27275 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.242879 27407 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.242842 25629 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.245322 27670 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.246069 29177 raft_consensus.cc:1050] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177: attempting to promote NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 to VOTER
I20240502 07:59:06.246500 28971 ts_tablet_manager.cc:902] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:59:06.246564 29177 consensus_queue.cc:237] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:06.247146 28971 tablet_copy_client.cc:308] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.132:45865
I20240502 07:59:06.247767 25759 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.248299 27407 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.248901 27538 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.249426 26712 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 07:59:06.249941 25343 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet ee25634bf534443b90fdbdcebdbcb02e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 6 but the committed config has opid_index of 7
I20240502 07:59:06.250370 25353 catalog_manager.cc:5484] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.251523 25349 catalog_manager.cc:5484] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 5 to 6, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:06.253525 25758 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.254278 26712 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.254828 25495 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.256376 25496 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.256371 27275 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 07:59:06.256603 25826 consensus_peers.cc:485] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0aa60608089c4f50bcee339d01815a65. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:06.257534 25911 tablet_copy_service.cc:140] P e9596bfb8d324b4da93a4288771ed177: Received BeginTabletCopySession request for tablet dbfd23aa383f413cbe4a5ccd8146d2b6 from peer a17bbff6a71b4667b14f5dd12bca3057 ({username='slave'} at 127.18.13.134:49677)
I20240502 07:59:06.257650 25911 tablet_copy_service.cc:161] P e9596bfb8d324b4da93a4288771ed177: Beginning new tablet copy session on tablet dbfd23aa383f413cbe4a5ccd8146d2b6 from peer a17bbff6a71b4667b14f5dd12bca3057 at {username='slave'} at 127.18.13.134:49677: session id = a17bbff6a71b4667b14f5dd12bca3057-dbfd23aa383f413cbe4a5ccd8146d2b6
I20240502 07:59:06.258733 25911 tablet_copy_source_session.cc:205] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:06.259981 29168 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.260483 25759 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.261744 25496 consensus_queue.cc:237] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:06.261791 29168 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.262089 29168 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.263716 27407 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.265141 27671 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.265811 29108 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.266564 27538 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.267107 29108 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.268429 25496 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.268465 27275 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.268657 25348 catalog_manager.cc:5484] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 6 to 7, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.269099 28375 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.270918 29168 raft_consensus.cc:1050] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177: attempting to promote NON_VOTER c5f3003b08fb4214b116c0be8dd62851 to VOTER
I20240502 07:59:06.271131 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ee25634bf534443b90fdbdcebdbcb02e with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.271471 29168 consensus_queue.cc:237] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:06.272235 27670 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.272423 28971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfd23aa383f413cbe4a5ccd8146d2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:06.272976 25341 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 03016ecfa11d4987a2e9d2368a66a5a2 with cas_config_opid_index 6: aborting the task: latest config opid_index 7; task opid_index 6
I20240502 07:59:06.273666 27275 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.274932 27407 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.275223 27538 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.275475 29177 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.277056 25759 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.277957 29177 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.278764 25888 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.278023 25354 catalog_manager.cc:5484] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 reported cstate change: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.279434 28971 tablet_copy_client.cc:783] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:06.279671 28971 tablet_copy_client.cc:647] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:06.279934 28375 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.280907 25888 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.281188 29177 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.281311 25496 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 07:59:06.281922 25826 consensus_peers.cc:485] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0aa60608089c4f50bcee339d01815a65. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:06.282698 25888 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.283990 29168 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.284937 26712 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.287264 27407 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.289222 25758 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.293030 27275 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.293860 25496 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.296239 25352 catalog_manager.cc:5484] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 reported cstate change: config changed from index 7 to 8, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
W20240502 07:59:06.297370 25342 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 0aa60608089c4f50bcee339d01815a65 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 7 but the committed config has opid_index of 8
I20240502 07:59:06.297358 25758 consensus_queue.cc:237] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:06.299469 27539 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.300091 29320 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.302668 27407 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.303718 25495 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.304246 25607 tablet_service.cc:1514] Processing DeleteTablet for tablet ee25634bf534443b90fdbdcebdbcb02e with delete_type TABLET_DATA_TOMBSTONED (TS 0dfa00c061c748b18772d5f0566944b5 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34854
I20240502 07:59:06.304246 29320 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.304502 29198 tablet_replica.cc:330] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: stopping tablet replica
I20240502 07:59:06.304649 29198 raft_consensus.cc:2229] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.304895 29198 raft_consensus.cc:2258] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.306310 29320 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.307745 29198 ts_tablet_manager.cc:1875] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.308446 25341 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 03016ecfa11d4987a2e9d2368a66a5a2 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.310528 25888 consensus_queue.cc:237] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:06.310586 25349 catalog_manager.cc:5484] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:06.311190 25495 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.311548 29198 ts_tablet_manager.cc:1888] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 07:59:06.312085 27670 raft_consensus.cc:1203] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:06.313007 27408 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.313458 26712 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.313644 29198 log.cc:1196] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/ee25634bf534443b90fdbdcebdbcb02e
I20240502 07:59:06.314280 25343 catalog_manager.cc:4830] TS 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): tablet ee25634bf534443b90fdbdcebdbcb02e (table rebalance_test_table_0 [id=616a1d4f85774bf6b08fc4da5a27702a]) successfully deleted
I20240502 07:59:06.315315 27274 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.315760 26712 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.317119 27670 raft_consensus.cc:1203] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:06.317243 28971 tablet_copy_client.cc:523] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:06.318063 25758 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.318652 26712 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.318976 28971 tablet_bootstrap.cc:492] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap starting.
I20240502 07:59:06.315315 27275 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.319746 29320 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.320868 27538 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.321683 26712 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.323853 25342 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0aa60608089c4f50bcee339d01815a65 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.324215 27274 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.324211 27408 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.324811 25355 catalog_manager.cc:5484] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 8 to 9, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:06.324911 25758 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.327041 25495 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.327765 29169 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.328902 25888 consensus_queue.cc:237] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:06.330489 27408 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:06.330907 29168 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:06.331445 27274 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:06.331620 25758 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:06.331964 29168 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:06.332090 29177 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:06.332535 25495 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.328478 27518 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa60608089c4f50bcee339d01815a65 with delete_type TABLET_DATA_TOMBSTONED (TS c6746058199446968d2496df55cdee42 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56612
I20240502 07:59:06.334822 25496 consensus_queue.cc:237] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:06.336043 27275 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.337400 29168 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.339087 27539 raft_consensus.cc:1203] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:06.339488 25758 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.339972 28374 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.340102 25342 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0aa60608089c4f50bcee339d01815a65 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.340404 25888 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.340823 27408 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.341075 28374 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.341225 28515 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.341169 25759 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.342156 27405 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.343650 28885 ts_tablet_manager.cc:902] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:06.344210 28885 tablet_copy_client.cc:308] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:06.344620 25760 consensus_queue.cc:237] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:06.344708 25517 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet d8e50f3d37d842868900a4cff5bf3601 from peer c6746058199446968d2496df55cdee42 ({username='slave'} at 127.18.13.135:42665)
I20240502 07:59:06.344812 25517 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet d8e50f3d37d842868900a4cff5bf3601 from peer c6746058199446968d2496df55cdee42 at {username='slave'} at 127.18.13.135:42665: session id = c6746058199446968d2496df55cdee42-d8e50f3d37d842868900a4cff5bf3601
I20240502 07:59:06.345906 25517 tablet_copy_source_session.cc:205] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:06.346740 28776 tablet_bootstrap.cc:492] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:06.347028 28776 tablet_bootstrap.cc:492] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851: Bootstrap complete.
I20240502 07:59:06.347222 28776 ts_tablet_manager.cc:1367] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851: Time spent bootstrapping tablet: real 0.378s user 0.022s sys 0.003s
I20240502 07:59:06.348096 28885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e50f3d37d842868900a4cff5bf3601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:06.347751 28776 raft_consensus.cc:348] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:06.348574 28776 raft_consensus.cc:729] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5f3003b08fb4214b116c0be8dd62851, State: Initialized, Role: LEARNER
I20240502 07:59:06.348698 28776 consensus_queue.cc:260] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:06.348819 27275 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.349509 28776 ts_tablet_manager.cc:1398] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 07:59:06.350047 25649 tablet_copy_service.cc:342] P 0dfa00c061c748b18772d5f0566944b5: Request end of tablet copy session c5f3003b08fb4214b116c0be8dd62851-fc1ac05112724e0ea1f84f648598304e received from {username='slave'} at 127.18.13.133:54317
I20240502 07:59:06.350126 25649 tablet_copy_service.cc:433] P 0dfa00c061c748b18772d5f0566944b5: ending tablet copy session c5f3003b08fb4214b116c0be8dd62851-fc1ac05112724e0ea1f84f648598304e on tablet fc1ac05112724e0ea1f84f648598304e with peer c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:06.350417 25607 tablet_service.cc:1514] Processing DeleteTablet for tablet 03016ecfa11d4987a2e9d2368a66a5a2 with delete_type TABLET_DATA_TOMBSTONED (TS 0dfa00c061c748b18772d5f0566944b5 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34854
I20240502 07:59:06.350625 28885 tablet_copy_client.cc:783] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:06.350801 28885 tablet_copy_client.cc:647] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:06.348112 27539 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.351236 25496 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.351178 25350 catalog_manager.cc:5484] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 9 to 10, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.351725 29320 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.352653 29198 tablet_replica.cc:330] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5: stopping tablet replica
I20240502 07:59:06.352653 29320 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.352803 29198 raft_consensus.cc:2229] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.355283 28374 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.355480 29320 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.356382 25888 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.358318 27539 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.358951 25341 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 03016ecfa11d4987a2e9d2368a66a5a2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.359601 25477 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa60608089c4f50bcee339d01815a65 with delete_type TABLET_DATA_TOMBSTONED (TS c3be11f0fe304e0f8e008dc1e4c41f70 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:50936
I20240502 07:59:06.359884 29300 tablet_replica.cc:330] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70: stopping tablet replica
I20240502 07:59:06.360029 29300 raft_consensus.cc:2229] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.360297 29300 raft_consensus.cc:2258] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.360752 25496 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.361971 29300 ts_tablet_manager.cc:1875] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.362934 27406 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.363287 29198 raft_consensus.cc:2258] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.364212 25888 consensus_queue.cc:237] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:06.365243 29108 raft_consensus.cc:1050] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 to VOTER
I20240502 07:59:06.365276 25352 catalog_manager.cc:5484] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 7 to 8. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.365641 29108 consensus_queue.cc:237] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:06.365746 25349 catalog_manager.cc:5484] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 8 to 9, VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) evicted. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:06.367579 25888 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:06.367761 25759 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:59:06.368026 28375 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:06.368232 26712 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:06.368705 29300 ts_tablet_manager.cc:1888] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 07:59:06.368788 29300 log.cc:1196] T 0aa60608089c4f50bcee339d01815a65 P c3be11f0fe304e0f8e008dc1e4c41f70: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/0aa60608089c4f50bcee339d01815a65
I20240502 07:59:06.369328 25343 catalog_manager.cc:4830] TS c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377): tablet 0aa60608089c4f50bcee339d01815a65 (table rebalance_test_table_4 [id=f72d1ad6e00b46cf99f485a9ef60b9b1]) successfully deleted
I20240502 07:59:06.370066 27670 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:06.370468 25629 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:06.370759 29198 ts_tablet_manager.cc:1875] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.371076 28515 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:06.371109 29169 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.371477 28515 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 07:59:06.372673 25342 catalog_manager.cc:4797] TS c6746058199446968d2496df55cdee42 (127.18.13.135:34615): delete failed for tablet 0aa60608089c4f50bcee339d01815a65 because the tablet was not found. No further retry: Not found: Tablet not found: 0aa60608089c4f50bcee339d01815a65
I20240502 07:59:06.373114 25759 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.373883 27406 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:59:06.375200 25758 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.377350 26712 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:06.378687 25496 consensus_queue.cc:237] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:06.378890 27406 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.380352 27405 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.380748 28515 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.381449 25758 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.381955 28375 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 07:59:06.382692 25433 consensus_peers.cc:485] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff31ddd6b3d74a9aae6617afb56c40f5. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:06.383262 25347 catalog_manager.cc:5484] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 10 to 11, VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) evicted. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:06.383893 29198 ts_tablet_manager.cc:1888] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 07:59:06.383981 29198 log.cc:1196] T 03016ecfa11d4987a2e9d2368a66a5a2 P 0dfa00c061c748b18772d5f0566944b5: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/03016ecfa11d4987a2e9d2368a66a5a2
I20240502 07:59:06.385524 28885 tablet_copy_client.cc:523] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:06.386272 25343 catalog_manager.cc:4830] TS 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): tablet 03016ecfa11d4987a2e9d2368a66a5a2 (table rebalance_test_table_3 [id=7c7fb9171c9548a7ad2d45e295d780cb]) successfully deleted
I20240502 07:59:06.387248 28885 tablet_bootstrap.cc:492] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42: Bootstrap starting.
I20240502 07:59:06.387995 27671 raft_consensus.cc:1203] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:06.388583 25342 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0aa60608089c4f50bcee339d01815a65 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.389003 25890 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.390367 28375 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.390978 28375 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.393990 27407 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.397503 25343 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ff31ddd6b3d74a9aae6617afb56c40f5 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:59:06.398677 27256 tablet_service.cc:1514] Processing DeleteTablet for tablet 03016ecfa11d4987a2e9d2368a66a5a2 with delete_type TABLET_DATA_TOMBSTONED (TS c5f3003b08fb4214b116c0be8dd62851 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40960
I20240502 07:59:06.399168 27255 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa60608089c4f50bcee339d01815a65 with delete_type TABLET_DATA_TOMBSTONED (TS c5f3003b08fb4214b116c0be8dd62851 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:40960
I20240502 07:59:06.399061 25347 catalog_manager.cc:5484] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 reported cstate change: config changed from index 8 to 9, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.399482 25758 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.401269 29108 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.403592 27671 raft_consensus.cc:1203] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:06.404091 25629 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.406584 28479 raft_consensus.cc:1050] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a: attempting to promote NON_VOTER c5f3003b08fb4214b116c0be8dd62851 to VOTER
I20240502 07:59:06.407248 28479 consensus_queue.cc:237] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:06.409413 27671 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.410077 29320 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.410408 25889 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.410938 25347 catalog_manager.cc:5484] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:06.412367 27275 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.412662 25888 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.413066 29320 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.413213 28880 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.414952 25629 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.415609 28880 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.418869 25890 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.420872 29411 tablet_replica.cc:330] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851: stopping tablet replica
I20240502 07:59:06.421790 29411 raft_consensus.cc:2229] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.422978 29411 raft_consensus.cc:2258] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.426298 29411 ts_tablet_manager.cc:1875] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.427006 25496 consensus_queue.cc:237] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:06.427273 27670 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.428467 27407 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:06.429064 28515 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:06.430994 29109 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.432880 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ff31ddd6b3d74a9aae6617afb56c40f5 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.433179 25888 raft_consensus.cc:1261] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 07:59:06.433492 25433 consensus_peers.cc:485] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff31ddd6b3d74a9aae6617afb56c40f5. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:06.433699 28375 consensus_queue.cc:1035] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:06.433670 27407 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.434672 25888 raft_consensus.cc:2932] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.436426 25354 catalog_manager.cc:5484] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 reported cstate change: config changed from index 9 to 10, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.440585 25738 tablet_service.cc:1514] Processing DeleteTablet for tablet ff31ddd6b3d74a9aae6617afb56c40f5 with delete_type TABLET_DATA_TOMBSTONED (TS 14d8fd1fc3a04d9d967b8033df84f34a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55602
I20240502 07:59:06.440848 29312 tablet_replica.cc:330] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a: stopping tablet replica
I20240502 07:59:06.440873 29411 ts_tablet_manager.cc:1888] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 07:59:06.440984 29411 log.cc:1196] T 03016ecfa11d4987a2e9d2368a66a5a2 P c5f3003b08fb4214b116c0be8dd62851: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal/wals/03016ecfa11d4987a2e9d2368a66a5a2
I20240502 07:59:06.440995 29312 raft_consensus.cc:2229] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.441213 29312 raft_consensus.cc:2258] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.442062 29411 tablet_replica.cc:330] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: stopping tablet replica
I20240502 07:59:06.442044 25343 catalog_manager.cc:4830] TS c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): tablet 03016ecfa11d4987a2e9d2368a66a5a2 (table rebalance_test_table_3 [id=7c7fb9171c9548a7ad2d45e295d780cb]) successfully deleted
I20240502 07:59:06.442189 29411 raft_consensus.cc:2229] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.442381 29312 ts_tablet_manager.cc:1875] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.442889 29411 raft_consensus.cc:2258] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.443758 29411 ts_tablet_manager.cc:1875] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.444784 29312 ts_tablet_manager.cc:1888] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 07:59:06.444854 29312 log.cc:1196] T ff31ddd6b3d74a9aae6617afb56c40f5 P 14d8fd1fc3a04d9d967b8033df84f34a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/ff31ddd6b3d74a9aae6617afb56c40f5
I20240502 07:59:06.445793 25341 catalog_manager.cc:4830] TS 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205): tablet ff31ddd6b3d74a9aae6617afb56c40f5 (table rebalance_test_table_0 [id=616a1d4f85774bf6b08fc4da5a27702a]) successfully deleted
I20240502 07:59:06.446210 29411 ts_tablet_manager.cc:1888] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 07:59:06.446283 29411 log.cc:1196] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal/wals/0aa60608089c4f50bcee339d01815a65
I20240502 07:59:06.447029 25888 consensus_queue.cc:237] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:06.449646 25343 catalog_manager.cc:4830] TS c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): tablet 0aa60608089c4f50bcee339d01815a65 (table rebalance_test_table_4 [id=f72d1ad6e00b46cf99f485a9ef60b9b1]) successfully deleted
I20240502 07:59:06.450250 25629 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:06.450980 25758 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 07:59:06.450986 25827 consensus_peers.cc:485] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 843d1c1d6384476983749a78f65a5c74. This is attempt 1: this message will repeat every 5th retry.
W20240502 07:59:06.451506 25824 consensus_peers.cc:485] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 843d1c1d6384476983749a78f65a5c74. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:06.451403 28880 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:06.451963 29352 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:06.453351 29352 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:06.454949 25888 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:06.455370 29321 raft_consensus.cc:1050] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a: attempting to promote NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 to VOTER
I20240502 07:59:06.455753 29321 consensus_queue.cc:237] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:06.456133 25347 catalog_manager.cc:5484] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:06.457846 27670 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.458274 27671 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:06.458727 27275 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:06.461859 25629 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:06.462297 28464 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.465607 25626 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.466944 28464 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.466841 29352 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:06.469208 25888 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.469707 27275 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.470288 28464 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.471333 25629 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:06.471557 25758 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:06.473423 28960 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.477149 28479 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.479610 27671 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.480309 25349 catalog_manager.cc:5484] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 4 to 5, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:06.487993 25888 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.492317 25354 catalog_manager.cc:5484] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38 reported cstate change: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.492488 27274 raft_consensus.cc:1203] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:06.502249 25888 consensus_queue.cc:237] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:06.502867 25347 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet a3bc416015dc40d88743215d2cf46230 with cas_config_opid_index 6: aborting the task: latest config opid_index 7; task opid_index 6
I20240502 07:59:06.504164 27407 raft_consensus.cc:1261] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.505093 29169 consensus_queue.cc:1035] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 07:59:06.505956 25826 consensus_peers.cc:485] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6929d2560404acb9bb53f54a181f958. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:06.506606 25758 consensus_queue.cc:237] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:06.506808 25496 raft_consensus.cc:1261] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.509011 27671 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.509441 27274 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.509418 29321 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.509744 26712 consensus_queue.cc:1035] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.510032 29321 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.511870 27274 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.517086 26712 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.520231 25888 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.521225 29321 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.522114 25347 catalog_manager.cc:5484] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:06.522284 25496 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.528671 29321 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.529464 25888 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.531994 27671 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.532336 27274 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.535037 25341 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3bc416015dc40d88743215d2cf46230 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.535570 25350 catalog_manager.cc:5484] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851 reported cstate change: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.539930 25607 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bc416015dc40d88743215d2cf46230 with delete_type TABLET_DATA_TOMBSTONED (TS 0dfa00c061c748b18772d5f0566944b5 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34854
I20240502 07:59:06.540208 29198 tablet_replica.cc:330] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5: stopping tablet replica
I20240502 07:59:06.540349 29198 raft_consensus.cc:2229] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.540511 29198 pending_rounds.cc:62] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5: Trying to abort 1 pending ops.
I20240502 07:59:06.540534 29198 pending_rounds.cc:68] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7023152113479430144 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "a3bc416015dc40d88743215d2cf46230" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } } }
I20240502 07:59:06.540683 29198 raft_consensus.cc:2894] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240502 07:59:06.540738 29198 raft_consensus.cc:2866] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 07:59:06.540776 29198 raft_consensus.cc:2258] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.541075 25758 consensus_queue.cc:237] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:06.543668 27671 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.544278 29320 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.545306 27274 raft_consensus.cc:1261] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.545857 29320 consensus_queue.cc:1035] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.546725 28885 tablet_bootstrap.cc:492] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:06.547152 28885 tablet_bootstrap.cc:492] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42: Bootstrap complete.
I20240502 07:59:06.547478 28885 ts_tablet_manager.cc:1367] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42: Time spent bootstrapping tablet: real 0.160s user 0.029s sys 0.000s
I20240502 07:59:06.548156 28885 raft_consensus.cc:348] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:06.548517 28960 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.549077 27671 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.550629 29198 ts_tablet_manager.cc:1875] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.551309 25341 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a3bc416015dc40d88743215d2cf46230 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.553754 29198 ts_tablet_manager.cc:1888] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 07:59:06.553954 29198 log.cc:1196] T a3bc416015dc40d88743215d2cf46230 P 0dfa00c061c748b18772d5f0566944b5: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/a3bc416015dc40d88743215d2cf46230
I20240502 07:59:06.554164 28885 raft_consensus.cc:729] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6746058199446968d2496df55cdee42, State: Initialized, Role: LEARNER
I20240502 07:59:06.554456 28885 consensus_queue.cc:260] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:06.554657 25343 catalog_manager.cc:4830] TS 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): tablet a3bc416015dc40d88743215d2cf46230 (table rebalance_test_table_5 [id=a5520c28ad46406f9fdf6855d5dfa37b]) successfully deleted
I20240502 07:59:06.556016 25347 catalog_manager.cc:5484] T a3bc416015dc40d88743215d2cf46230 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 8 to 9, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:06.559439 27407 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.561342 28885 ts_tablet_manager.cc:1398] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240502 07:59:06.563239 27274 raft_consensus.cc:2932] T a3bc416015dc40d88743215d2cf46230 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.564935 25517 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session c6746058199446968d2496df55cdee42-d8e50f3d37d842868900a4cff5bf3601 received from {username='slave'} at 127.18.13.135:42665
I20240502 07:59:06.565032 25517 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session c6746058199446968d2496df55cdee42-d8e50f3d37d842868900a4cff5bf3601 on tablet d8e50f3d37d842868900a4cff5bf3601 with peer c6746058199446968d2496df55cdee42
I20240502 07:59:06.567226 25870 tablet_service.cc:1514] Processing DeleteTablet for tablet a3bc416015dc40d88743215d2cf46230 with delete_type TABLET_DATA_TOMBSTONED (TS e9596bfb8d324b4da93a4288771ed177 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47880
I20240502 07:59:06.567509 29229 tablet_replica.cc:330] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177: stopping tablet replica
I20240502 07:59:06.567683 29229 raft_consensus.cc:2229] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.567945 29229 raft_consensus.cc:2258] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.569043 29229 ts_tablet_manager.cc:1875] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.571568 29229 ts_tablet_manager.cc:1888] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 07:59:06.571655 29229 log.cc:1196] T a3bc416015dc40d88743215d2cf46230 P e9596bfb8d324b4da93a4288771ed177: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/wals/a3bc416015dc40d88743215d2cf46230
I20240502 07:59:06.577414 25342 catalog_manager.cc:4830] TS e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865): tablet a3bc416015dc40d88743215d2cf46230 (table rebalance_test_table_5 [id=a5520c28ad46406f9fdf6855d5dfa37b]) successfully deleted
I20240502 07:59:06.586316 25496 consensus_queue.cc:237] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
W20240502 07:59:06.588531 25432 consensus_peers.cc:485] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb0c33c3e4014eb3b931d64adac60e1a. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:06.588737 27274 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.589159 25626 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.589759 28375 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.590649 29109 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.593753 28375 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:06.598534 27274 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:06.598524 25629 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:06.600355 29320 raft_consensus.cc:1050] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a: attempting to promote NON_VOTER c5f3003b08fb4214b116c0be8dd62851 to VOTER
I20240502 07:59:06.600947 29320 consensus_queue.cc:237] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:06.603888 27671 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.604007 25888 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.603449 25354 catalog_manager.cc:5484] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 6 to 7, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:06.604499 28960 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.604800 25629 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.605307 28960 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.606333 27274 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.607333 27589 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.607684 27589 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.617506 28971 tablet_bootstrap.cc:492] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:06.617858 28971 tablet_bootstrap.cc:492] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap complete.
I20240502 07:59:06.618315 27589 raft_consensus.cc:1013] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: attempt to promote peer 85fbc6c98f0643469c92215a5cfb2e38: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 07:59:06.625699 28960 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.628327 25629 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.629012 27671 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.629201 27274 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.631902 28971 ts_tablet_manager.cc:1367] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057: Time spent bootstrapping tablet: real 0.313s user 0.031s sys 0.001s
I20240502 07:59:06.632519 28971 raft_consensus.cc:348] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:06.632756 25354 catalog_manager.cc:5484] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.633189 29320 raft_consensus.cc:1050] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a: attempting to promote NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 to VOTER
I20240502 07:59:06.633664 29320 consensus_queue.cc:237] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:06.635627 25629 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.635704 25888 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.636209 28960 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.636261 27671 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.637529 28479 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.637670 27274 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.638178 28479 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.638819 28479 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.638890 28971 raft_consensus.cc:729] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a17bbff6a71b4667b14f5dd12bca3057, State: Initialized, Role: LEARNER
I20240502 07:59:06.638923 25888 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.639086 28971 consensus_queue.cc:260] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:06.640154 28971 ts_tablet_manager.cc:1398] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240502 07:59:06.640646 25911 tablet_copy_service.cc:342] P e9596bfb8d324b4da93a4288771ed177: Request end of tablet copy session a17bbff6a71b4667b14f5dd12bca3057-dbfd23aa383f413cbe4a5ccd8146d2b6 received from {username='slave'} at 127.18.13.134:49677
I20240502 07:59:06.640735 25911 tablet_copy_service.cc:433] P e9596bfb8d324b4da93a4288771ed177: ending tablet copy session a17bbff6a71b4667b14f5dd12bca3057-dbfd23aa383f413cbe4a5ccd8146d2b6 on tablet dbfd23aa383f413cbe4a5ccd8146d2b6 with peer a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:06.658609 25496 consensus_queue.cc:237] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:06.658895 28479 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.660938 25629 raft_consensus.cc:1261] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:06.661547 28374 consensus_queue.cc:1035] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:06.661881 27274 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.664655 28515 raft_consensus.cc:2932] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.666297 25758 raft_consensus.cc:1261] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:59:06.666457 25433 consensus_peers.cc:485] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16a130f60be74f568cc53a70692a6838. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:06.666610 25354 catalog_manager.cc:5484] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851 reported cstate change: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.666795 29108 consensus_queue.cc:1035] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:06.668022 27671 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.668326 25626 raft_consensus.cc:2932] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.667614 25629 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.670727 25888 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.672291 25758 consensus_queue.cc:237] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:06.673014 25759 raft_consensus.cc:2932] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.673532 25349 catalog_manager.cc:5484] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index -1 to 4, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.673792 27671 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.674253 27274 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.674757 27589 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.677947 28464 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.679912 25629 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.680480 28464 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.682921 28464 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.683363 27274 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.686754 27671 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.689292 25341 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 273e203d4ed44c43859f08e9bec08a9d with cas_config_opid_index 6: aborting the task: latest config opid_index 7; task opid_index 6
I20240502 07:59:06.690026 25352 catalog_manager.cc:5484] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38 reported cstate change: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.690052 25341 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 273e203d4ed44c43859f08e9bec08a9d with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.695950 25870 tablet_service.cc:1514] Processing DeleteTablet for tablet 273e203d4ed44c43859f08e9bec08a9d with delete_type TABLET_DATA_TOMBSTONED (TS e9596bfb8d324b4da93a4288771ed177 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47880
I20240502 07:59:06.696228 29229 tablet_replica.cc:330] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177: stopping tablet replica
I20240502 07:59:06.696382 29229 raft_consensus.cc:2229] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.696609 29229 raft_consensus.cc:2258] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.697863 29229 ts_tablet_manager.cc:1875] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.700280 29229 ts_tablet_manager.cc:1888] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 07:59:06.700369 29229 log.cc:1196] T 273e203d4ed44c43859f08e9bec08a9d P e9596bfb8d324b4da93a4288771ed177: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/wals/273e203d4ed44c43859f08e9bec08a9d
I20240502 07:59:06.701198 25759 consensus_queue.cc:237] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:06.702702 27671 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.702731 27274 raft_consensus.cc:1261] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.703155 25342 catalog_manager.cc:4830] TS e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865): tablet 273e203d4ed44c43859f08e9bec08a9d (table rebalance_test_table_7 [id=89a9dd84e8b5432a81ef7767804381d3]) successfully deleted
I20240502 07:59:06.703231 29320 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.703475 28464 consensus_queue.cc:1035] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.705417 29320 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.706964 27274 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.707868 27671 raft_consensus.cc:2932] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:06.709944 25341 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 273e203d4ed44c43859f08e9bec08a9d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.712548 25348 catalog_manager.cc:5484] T 273e203d4ed44c43859f08e9bec08a9d P 85fbc6c98f0643469c92215a5cfb2e38 reported cstate change: config changed from index 8 to 9, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
W20240502 07:59:06.714507 25341 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 273e203d4ed44c43859f08e9bec08a9d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER failed with leader 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 8 but the committed config has opid_index of 9
I20240502 07:59:06.715644 25607 tablet_service.cc:1514] Processing DeleteTablet for tablet 273e203d4ed44c43859f08e9bec08a9d with delete_type TABLET_DATA_TOMBSTONED (TS 0dfa00c061c748b18772d5f0566944b5 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34854
I20240502 07:59:06.715951 29198 tablet_replica.cc:330] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5: stopping tablet replica
I20240502 07:59:06.716100 29198 raft_consensus.cc:2229] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.716296 29198 pending_rounds.cc:62] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5: Trying to abort 1 pending ops.
I20240502 07:59:06.716331 29198 pending_rounds.cc:68] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023152114368397312 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "273e203d4ed44c43859f08e9bec08a9d" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } } }
I20240502 07:59:06.716516 29198 raft_consensus.cc:2894] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240502 07:59:06.716598 29198 raft_consensus.cc:2866] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 07:59:06.716650 29198 raft_consensus.cc:2258] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.718794 29198 ts_tablet_manager.cc:1875] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.721428 29198 ts_tablet_manager.cc:1888] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 07:59:06.721524 29198 log.cc:1196] T 273e203d4ed44c43859f08e9bec08a9d P 0dfa00c061c748b18772d5f0566944b5: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/273e203d4ed44c43859f08e9bec08a9d
I20240502 07:59:06.722332 25343 catalog_manager.cc:4830] TS 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): tablet 273e203d4ed44c43859f08e9bec08a9d (table rebalance_test_table_7 [id=89a9dd84e8b5432a81ef7767804381d3]) successfully deleted
I20240502 07:59:06.725979 27407 raft_consensus.cc:1203] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:06.733260 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:06.737318 27255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:06.737771 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:06.740607 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:06.741292 29109 raft_consensus.cc:1050] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 to VOTER
I20240502 07:59:06.741817 29109 consensus_queue.cc:237] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:06.743386 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:06.743739 27539 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.744158 25888 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.745088 29108 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.745410 29108 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.748497 27405 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.749086 28375 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.752934 28375 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.755378 25890 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.757978 25759 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.758508 29108 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.760182 27405 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.761919 25352 catalog_manager.cc:5484] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.763779 25890 consensus_queue.cc:237] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:06.767424 25496 raft_consensus.cc:1261] T c95cd9642c1c46cabb23f6c62c6ad4f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:06.767568 25758 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:06.768044 26712 consensus_queue.cc:1035] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:06.769114 29109 raft_consensus.cc:1050] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER c6746058199446968d2496df55cdee42 to VOTER
I20240502 07:59:06.769554 29109 consensus_queue.cc:237] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:06.770516 25629 raft_consensus.cc:1261] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:06.771816 26712 consensus_queue.cc:1035] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:06.773877 27538 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.774658 29108 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.775348 27539 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
W20240502 07:59:06.783733 25824 consensus_peers.cc:485] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c95cd9642c1c46cabb23f6c62c6ad4f5. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:06.785295 27407 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.788137 29108 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.791306 25890 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.791944 27518 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:06.791929 29108 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.793447 25758 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.793898 29108 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.797215 29108 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.798087 27539 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.800482 25890 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.802315 25758 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.803674 27405 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.805295 25349 catalog_manager.cc:5484] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42 reported cstate change: config changed from index 6 to 7, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.809713 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:06.821200 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:06.826026 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:06.829663 26712 raft_consensus.cc:2932] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:06.831017 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:06.832599 25626 raft_consensus.cc:2932] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:06.833813 25496 raft_consensus.cc:2932] T c95cd9642c1c46cabb23f6c62c6ad4f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:06.835211 25352 catalog_manager.cc:5484] T c95cd9642c1c46cabb23f6c62c6ad4f5 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:06.845566 28885 ts_tablet_manager.cc:902] T ff31ddd6b3d74a9aae6617afb56c40f5 P c6746058199446968d2496df55cdee42: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:06.846088 28885 tablet_copy_client.cc:308] T ff31ddd6b3d74a9aae6617afb56c40f5 P c6746058199446968d2496df55cdee42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:06.846590 25517 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet ff31ddd6b3d74a9aae6617afb56c40f5 from peer c6746058199446968d2496df55cdee42 ({username='slave'} at 127.18.13.135:42665)
I20240502 07:59:06.846745 25517 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet ff31ddd6b3d74a9aae6617afb56c40f5 from peer c6746058199446968d2496df55cdee42 at {username='slave'} at 127.18.13.135:42665: session id = c6746058199446968d2496df55cdee42-ff31ddd6b3d74a9aae6617afb56c40f5
I20240502 07:59:06.847755 25517 tablet_copy_source_session.cc:205] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:06.848870 28885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff31ddd6b3d74a9aae6617afb56c40f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:06.851372 28885 tablet_copy_client.cc:783] T ff31ddd6b3d74a9aae6617afb56c40f5 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:06.851528 28885 tablet_copy_client.cc:647] T ff31ddd6b3d74a9aae6617afb56c40f5 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:06.851886 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:06.858727 27255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:06.871984 29255 raft_consensus.cc:1050] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5: attempting to promote NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 to VOTER
I20240502 07:59:06.872444 28464 raft_consensus.cc:1050] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a: attempting to promote NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 to VOTER
I20240502 07:59:06.872536 29255 consensus_queue.cc:237] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:06.872840 28464 consensus_queue.cc:237] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:06.874162 27671 raft_consensus.cc:1261] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:06.874523 29321 consensus_queue.cc:1035] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:06.874735 25759 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.875268 27671 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.875809 29256 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.876170 25495 raft_consensus.cc:1261] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:06.876277 27539 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.876667 28464 consensus_queue.cc:1035] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:06.876786 27181 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.877074 27181 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.878171 25496 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:06.878607 29256 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:06.883733 25626 raft_consensus.cc:1261] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:06.885434 28885 tablet_copy_client.cc:523] T ff31ddd6b3d74a9aae6617afb56c40f5 P c6746058199446968d2496df55cdee42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:06.886986 28885 tablet_bootstrap.cc:492] T ff31ddd6b3d74a9aae6617afb56c40f5 P c6746058199446968d2496df55cdee42: Bootstrap starting.
I20240502 07:59:06.892078 28464 consensus_queue.cc:1035] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:06.895452 29587 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.897800 25759 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.900862 27671 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.901911 25496 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.903663 25348 catalog_manager.cc:5484] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38 reported cstate change: config changed from index 5 to 6, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.905206 25496 consensus_queue.cc:237] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:06.909739 25629 consensus_queue.cc:237] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:06.906807 27538 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.910935 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:06.911116 29108 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.912026 27671 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.912310 25759 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.912565 29255 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.913340 29255 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.913435 27405 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.914454 29109 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.916353 25496 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:06.916749 29587 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:06.916947 29588 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.917465 29109 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.918294 27405 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.919529 25759 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.919634 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1ad22d2e464d4defab6bd0061e6058f8 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.919950 28515 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.920325 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d8e50f3d37d842868900a4cff5bf3601 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.920529 27671 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.920616 25759 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.922338 25356 catalog_manager.cc:5484] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057 reported cstate change: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.922446 27538 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.923058 25349 catalog_manager.cc:5484] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:06.927413 25496 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.930589 25758 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.931128 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:06.932778 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:06.933434 25870 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e50f3d37d842868900a4cff5bf3601 with delete_type TABLET_DATA_TOMBSTONED (TS e9596bfb8d324b4da93a4288771ed177 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47880
I20240502 07:59:06.934165 29229 tablet_replica.cc:330] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177: stopping tablet replica
I20240502 07:59:06.934336 29229 raft_consensus.cc:2229] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:06.934514 29229 raft_consensus.cc:2258] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:06.934976 27518 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:06.936453 29229 ts_tablet_manager.cc:1875] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.938830 29229 ts_tablet_manager.cc:1888] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 07:59:06.938897 29229 log.cc:1196] T d8e50f3d37d842868900a4cff5bf3601 P e9596bfb8d324b4da93a4288771ed177: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/wals/d8e50f3d37d842868900a4cff5bf3601
I20240502 07:59:06.944139 28971 ts_tablet_manager.cc:902] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:59:06.944537 25342 catalog_manager.cc:4830] TS e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865): tablet d8e50f3d37d842868900a4cff5bf3601 (table rebalance_test_table_7 [id=89a9dd84e8b5432a81ef7767804381d3]) successfully deleted
I20240502 07:59:06.944619 28971 tablet_copy_client.cc:308] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.132:45865
I20240502 07:59:06.945567 25911 tablet_copy_service.cc:140] P e9596bfb8d324b4da93a4288771ed177: Received BeginTabletCopySession request for tablet 843d1c1d6384476983749a78f65a5c74 from peer a17bbff6a71b4667b14f5dd12bca3057 ({username='slave'} at 127.18.13.134:49677)
I20240502 07:59:06.945669 25911 tablet_copy_service.cc:161] P e9596bfb8d324b4da93a4288771ed177: Beginning new tablet copy session on tablet 843d1c1d6384476983749a78f65a5c74 from peer a17bbff6a71b4667b14f5dd12bca3057 at {username='slave'} at 127.18.13.134:49677: session id = a17bbff6a71b4667b14f5dd12bca3057-843d1c1d6384476983749a78f65a5c74
I20240502 07:59:06.946609 25911 tablet_copy_source_session.cc:205] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:06.947898 28971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843d1c1d6384476983749a78f65a5c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:06.950476 27518 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad22d2e464d4defab6bd0061e6058f8 with delete_type TABLET_DATA_TOMBSTONED (TS c6746058199446968d2496df55cdee42 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56612
I20240502 07:59:06.950726 25496 consensus_queue.cc:237] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:06.951294 28971 tablet_copy_client.cc:783] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:06.951462 28971 tablet_copy_client.cc:647] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:06.952538 27538 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.953034 29108 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.954264 25629 consensus_queue.cc:237] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:06.955528 28374 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.956565 27538 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.956892 27671 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.957662 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d8e50f3d37d842868900a4cff5bf3601 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.957922 29255 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.959748 27405 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:06.960309 28515 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:06.962100 27405 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.963455 25352 catalog_manager.cc:5484] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42 reported cstate change: config changed from index 8 to 9, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:06.966710 25738 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e50f3d37d842868900a4cff5bf3601 with delete_type TABLET_DATA_TOMBSTONED (TS 14d8fd1fc3a04d9d967b8033df84f34a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55602
I20240502 07:59:06.980134 29617 ts_tablet_manager.cc:902] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42: Initiating tablet copy from peer e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:59:06.982695 29255 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.983649 29602 tablet_replica.cc:330] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: stopping tablet replica
I20240502 07:59:06.983826 29602 raft_consensus.cc:2229] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 07:59:06.983934 29602 pending_rounds.cc:62] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: Trying to abort 1 pending ops.
I20240502 07:59:06.983960 29602 pending_rounds.cc:68] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023152115188088832 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "1ad22d2e464d4defab6bd0061e6058f8" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } } }
I20240502 07:59:06.984099 29602 raft_consensus.cc:2894] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240502 07:59:06.984158 29602 raft_consensus.cc:2866] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 07:59:06.984197 29602 raft_consensus.cc:2258] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 07:59:06.984071 29321 raft_consensus.cc:2932] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.984992 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1ad22d2e464d4defab6bd0061e6058f8 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:06.985227 29602 ts_tablet_manager.cc:1875] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:06.986888 27671 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.987370 25352 catalog_manager.cc:5484] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 7 to 8, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:06.988032 25349 catalog_manager.cc:5484] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:06.988020 25495 raft_consensus.cc:2932] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:06.989253 25758 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:06.992302 29602 ts_tablet_manager.cc:1888] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 07:59:06.993882 29602 log.cc:1196] T 1ad22d2e464d4defab6bd0061e6058f8 P c6746058199446968d2496df55cdee42: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-6/wal/wals/1ad22d2e464d4defab6bd0061e6058f8
I20240502 07:59:06.994792 25342 catalog_manager.cc:4830] TS c6746058199446968d2496df55cdee42 (127.18.13.135:34615): tablet 1ad22d2e464d4defab6bd0061e6058f8 (table rebalance_test_table_8 [id=8aaedf9499044a93ab9419f10e2342e2]) successfully deleted
I20240502 07:59:06.995465 25477 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ad22d2e464d4defab6bd0061e6058f8 with delete_type TABLET_DATA_TOMBSTONED (TS c3be11f0fe304e0f8e008dc1e4c41f70 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:50936
I20240502 07:59:06.996459 29588 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:06.997291 27671 raft_consensus.cc:2932] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:07.003875 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.005954 25758 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:07.011169 25758 consensus_queue.cc:237] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:07.011328 25626 raft_consensus.cc:2932] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:07.011934 29617 tablet_copy_client.cc:308] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.132:45865
I20240502 07:59:07.015830 28971 tablet_copy_client.cc:523] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:07.017527 28971 tablet_bootstrap.cc:492] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap starting.
I20240502 07:59:07.020370 25626 raft_consensus.cc:1261] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:07.020740 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.020977 29320 consensus_queue.cc:1035] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.022505 27671 raft_consensus.cc:1261] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:07.022869 29320 consensus_queue.cc:1035] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.023501 29624 tablet_replica.cc:330] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70: stopping tablet replica
I20240502 07:59:07.023725 29624 raft_consensus.cc:2229] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:07.024062 29624 raft_consensus.cc:2258] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:07.024873 29624 ts_tablet_manager.cc:1875] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:07.025808 29320 raft_consensus.cc:2932] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:07.027629 25341 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99c3ddedadbf4c0e8858085b66022342 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:07.028012 29624 ts_tablet_manager.cc:1888] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 07:59:07.028151 29624 log.cc:1196] T 1ad22d2e464d4defab6bd0061e6058f8 P c3be11f0fe304e0f8e008dc1e4c41f70: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/1ad22d2e464d4defab6bd0061e6058f8
I20240502 07:59:07.028883 25343 catalog_manager.cc:4830] TS c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377): tablet 1ad22d2e464d4defab6bd0061e6058f8 (table rebalance_test_table_8 [id=8aaedf9499044a93ab9419f10e2342e2]) successfully deleted
I20240502 07:59:07.029448 27670 raft_consensus.cc:2932] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:07.032223 25348 catalog_manager.cc:5484] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 5 to 6, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:07.035404 25477 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c3ddedadbf4c0e8858085b66022342 with delete_type TABLET_DATA_TOMBSTONED (TS c3be11f0fe304e0f8e008dc1e4c41f70 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50936
I20240502 07:59:07.035925 29624 tablet_replica.cc:330] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70: stopping tablet replica
I20240502 07:59:07.036180 29624 raft_consensus.cc:2229] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:07.036392 29624 raft_consensus.cc:2258] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:07.038161 29624 ts_tablet_manager.cc:1875] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:07.039261 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:07.039187 25626 raft_consensus.cc:2932] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:07.040742 29624 ts_tablet_manager.cc:1888] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:07.040884 29624 log.cc:1196] T 99c3ddedadbf4c0e8858085b66022342 P c3be11f0fe304e0f8e008dc1e4c41f70: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/99c3ddedadbf4c0e8858085b66022342
I20240502 07:59:07.041672 25343 catalog_manager.cc:4830] TS c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377): tablet 99c3ddedadbf4c0e8858085b66022342 (table rebalance_test_table_6 [id=67841c5c645f4e86970122328fd1238e]) successfully deleted
I20240502 07:59:07.043901 29611 ts_tablet_manager.cc:902] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851: Initiating tablet copy from peer e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:59:07.044214 29612 tablet_replica.cc:330] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a: stopping tablet replica
I20240502 07:59:07.044566 29612 raft_consensus.cc:2229] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:07.044816 29612 raft_consensus.cc:2258] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:07.048554 29612 ts_tablet_manager.cc:1875] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:07.054775 29612 ts_tablet_manager.cc:1888] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 07:59:07.056030 29612 log.cc:1196] T d8e50f3d37d842868900a4cff5bf3601 P 14d8fd1fc3a04d9d967b8033df84f34a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/d8e50f3d37d842868900a4cff5bf3601
I20240502 07:59:07.057138 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:07.058372 25341 catalog_manager.cc:4830] TS 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205): tablet d8e50f3d37d842868900a4cff5bf3601 (table rebalance_test_table_7 [id=89a9dd84e8b5432a81ef7767804381d3]) successfully deleted
I20240502 07:59:07.067927 29611 tablet_copy_client.cc:308] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.132:45865
I20240502 07:59:07.068511 25911 tablet_copy_service.cc:140] P e9596bfb8d324b4da93a4288771ed177: Received BeginTabletCopySession request for tablet 843d1c1d6384476983749a78f65a5c74 from peer c5f3003b08fb4214b116c0be8dd62851 ({username='slave'} at 127.18.13.133:39133)
I20240502 07:59:07.068646 25911 tablet_copy_service.cc:161] P e9596bfb8d324b4da93a4288771ed177: Beginning new tablet copy session on tablet 843d1c1d6384476983749a78f65a5c74 from peer c5f3003b08fb4214b116c0be8dd62851 at {username='slave'} at 127.18.13.133:39133: session id = c5f3003b08fb4214b116c0be8dd62851-843d1c1d6384476983749a78f65a5c74
I20240502 07:59:07.069581 25911 tablet_copy_source_session.cc:205] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:07.070572 29611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843d1c1d6384476983749a78f65a5c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:07.073164 29611 tablet_copy_client.cc:783] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:07.073470 25911 tablet_copy_service.cc:140] P e9596bfb8d324b4da93a4288771ed177: Received BeginTabletCopySession request for tablet d6929d2560404acb9bb53f54a181f958 from peer c6746058199446968d2496df55cdee42 ({username='slave'} at 127.18.13.135:50611)
I20240502 07:59:07.073562 25911 tablet_copy_service.cc:161] P e9596bfb8d324b4da93a4288771ed177: Beginning new tablet copy session on tablet d6929d2560404acb9bb53f54a181f958 from peer c6746058199446968d2496df55cdee42 at {username='slave'} at 127.18.13.135:50611: session id = c6746058199446968d2496df55cdee42-d6929d2560404acb9bb53f54a181f958
I20240502 07:59:07.074420 25911 tablet_copy_source_session.cc:205] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:07.075026 29611 tablet_copy_client.cc:647] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:07.075366 29617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6929d2560404acb9bb53f54a181f958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:07.076189 29611 tablet_copy_client.cc:523] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:07.077886 29611 tablet_bootstrap.cc:492] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap starting.
I20240502 07:59:07.078078 29617 tablet_copy_client.cc:783] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:07.078263 29617 tablet_copy_client.cc:647] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:07.084203 29617 tablet_copy_client.cc:523] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:07.085918 29617 tablet_bootstrap.cc:492] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42: Bootstrap starting.
I20240502 07:59:07.089884 27255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.098678 29588 raft_consensus.cc:1050] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5: attempting to promote NON_VOTER c5f3003b08fb4214b116c0be8dd62851 to VOTER
I20240502 07:59:07.099120 29588 consensus_queue.cc:237] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:07.100606 27518 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.101478 27274 raft_consensus.cc:1261] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:07.101866 29587 consensus_queue.cc:1035] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:07.104359 25758 raft_consensus.cc:1261] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:07.105317 25890 raft_consensus.cc:1261] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:07.106318 29587 consensus_queue.cc:1035] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:07.106516 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:07.113204 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:07.125226 25477 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 07:59:07.125787 29587 consensus_queue.cc:1035] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:07.127185 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:07.128639 29611 tablet_bootstrap.cc:492] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:07.128980 29611 tablet_bootstrap.cc:492] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap complete.
I20240502 07:59:07.129161 29611 ts_tablet_manager.cc:1367] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851: Time spent bootstrapping tablet: real 0.051s user 0.023s sys 0.002s
I20240502 07:59:07.129645 29611 raft_consensus.cc:348] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.129783 29611 raft_consensus.cc:729] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5f3003b08fb4214b116c0be8dd62851, State: Initialized, Role: LEARNER
I20240502 07:59:07.129885 29611 consensus_queue.cc:260] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.130985 29611 ts_tablet_manager.cc:1398] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:07.132305 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:07.132474 25911 tablet_copy_service.cc:342] P e9596bfb8d324b4da93a4288771ed177: Request end of tablet copy session c5f3003b08fb4214b116c0be8dd62851-843d1c1d6384476983749a78f65a5c74 received from {username='slave'} at 127.18.13.133:39133
I20240502 07:59:07.132534 25911 tablet_copy_service.cc:433] P e9596bfb8d324b4da93a4288771ed177: ending tablet copy session c5f3003b08fb4214b116c0be8dd62851-843d1c1d6384476983749a78f65a5c74 on tablet 843d1c1d6384476983749a78f65a5c74 with peer c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:07.158191 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.159082 28885 tablet_bootstrap.cc:492] T ff31ddd6b3d74a9aae6617afb56c40f5 P c6746058199446968d2496df55cdee42: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:07.159386 28885 tablet_bootstrap.cc:492] T ff31ddd6b3d74a9aae6617afb56c40f5 P c6746058199446968d2496df55cdee42: Bootstrap complete.
I20240502 07:59:07.159559 28885 ts_tablet_manager.cc:1367] T ff31ddd6b3d74a9aae6617afb56c40f5 P c6746058199446968d2496df55cdee42: Time spent bootstrapping tablet: real 0.273s user 0.027s sys 0.004s
I20240502 07:59:07.161216 28971 tablet_bootstrap.cc:492] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:07.161522 28971 tablet_bootstrap.cc:492] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap complete.
I20240502 07:59:07.161739 28971 ts_tablet_manager.cc:1367] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057: Time spent bootstrapping tablet: real 0.144s user 0.024s sys 0.003s
I20240502 07:59:07.162232 28971 raft_consensus.cc:348] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.162375 28971 raft_consensus.cc:729] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a17bbff6a71b4667b14f5dd12bca3057, State: Initialized, Role: LEARNER
I20240502 07:59:07.162489 28971 consensus_queue.cc:260] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.162986 28885 raft_consensus.cc:348] T ff31ddd6b3d74a9aae6617afb56c40f5 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:07.163125 28885 raft_consensus.cc:729] T ff31ddd6b3d74a9aae6617afb56c40f5 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6746058199446968d2496df55cdee42, State: Initialized, Role: LEARNER
I20240502 07:59:07.163228 28885 consensus_queue.cc:260] T ff31ddd6b3d74a9aae6617afb56c40f5 P c6746058199446968d2496df55cdee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:07.163515 28885 ts_tablet_manager.cc:1398] T ff31ddd6b3d74a9aae6617afb56c40f5 P c6746058199446968d2496df55cdee42: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 07:59:07.164651 25517 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session c6746058199446968d2496df55cdee42-ff31ddd6b3d74a9aae6617afb56c40f5 received from {username='slave'} at 127.18.13.135:42665
I20240502 07:59:07.164753 25517 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session c6746058199446968d2496df55cdee42-ff31ddd6b3d74a9aae6617afb56c40f5 on tablet ff31ddd6b3d74a9aae6617afb56c40f5 with peer c6746058199446968d2496df55cdee42
I20240502 07:59:07.167080 28971 ts_tablet_manager.cc:1398] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 07:59:07.168856 27518 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.173668 25911 tablet_copy_service.cc:342] P e9596bfb8d324b4da93a4288771ed177: Request end of tablet copy session a17bbff6a71b4667b14f5dd12bca3057-843d1c1d6384476983749a78f65a5c74 received from {username='slave'} at 127.18.13.134:49677
I20240502 07:59:07.173755 25911 tablet_copy_service.cc:433] P e9596bfb8d324b4da93a4288771ed177: ending tablet copy session a17bbff6a71b4667b14f5dd12bca3057-843d1c1d6384476983749a78f65a5c74 on tablet 843d1c1d6384476983749a78f65a5c74 with peer a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:07.176512 25870 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:07.187460 27255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.193811 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.196874 29587 raft_consensus.cc:2932] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:07.200897 25890 raft_consensus.cc:2932] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:07.201011 25348 catalog_manager.cc:5484] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 4 to 5, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:07.203919 25758 raft_consensus.cc:2932] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:07.206164 27274 raft_consensus.cc:2932] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:07.210265 25626 consensus_queue.cc:237] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:07.211508 27274 raft_consensus.cc:1261] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:07.211970 29256 consensus_queue.cc:1035] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.213511 25758 raft_consensus.cc:1261] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:07.213862 29256 consensus_queue.cc:1035] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.215085 28885 ts_tablet_manager.cc:902] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:07.215615 28885 tablet_copy_client.cc:308] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:07.216327 25517 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet 16a130f60be74f568cc53a70692a6838 from peer c6746058199446968d2496df55cdee42 ({username='slave'} at 127.18.13.135:42665)
I20240502 07:59:07.216435 25517 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet 16a130f60be74f568cc53a70692a6838 from peer c6746058199446968d2496df55cdee42 at {username='slave'} at 127.18.13.135:42665: session id = c6746058199446968d2496df55cdee42-16a130f60be74f568cc53a70692a6838
I20240502 07:59:07.217015 29666 ts_tablet_manager.cc:902] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:07.217469 29666 tablet_copy_client.cc:308] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:07.217494 25517 tablet_copy_source_session.cc:205] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:07.218488 28885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16a130f60be74f568cc53a70692a6838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:07.220389 29256 raft_consensus.cc:2932] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:07.220836 28885 tablet_copy_client.cc:783] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:07.220978 28885 tablet_copy_client.cc:647] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:07.221853 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fc1ac05112724e0ea1f84f648598304e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:07.221972 25758 raft_consensus.cc:2932] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:07.222249 27274 raft_consensus.cc:2932] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:07.223457 25517 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet eb0c33c3e4014eb3b931d64adac60e1a from peer 85fbc6c98f0643469c92215a5cfb2e38 ({username='slave'} at 127.18.13.136:58689)
I20240502 07:59:07.223553 25517 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet eb0c33c3e4014eb3b931d64adac60e1a from peer 85fbc6c98f0643469c92215a5cfb2e38 at {username='slave'} at 127.18.13.136:58689: session id = 85fbc6c98f0643469c92215a5cfb2e38-eb0c33c3e4014eb3b931d64adac60e1a
I20240502 07:59:07.224566 25517 tablet_copy_source_session.cc:205] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:07.224460 25352 catalog_manager.cc:5484] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 5 to 6, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:07.227571 29666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0c33c3e4014eb3b931d64adac60e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:07.229903 29666 tablet_copy_client.cc:783] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:07.230041 29666 tablet_copy_client.cc:647] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:07.230172 25870 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1ac05112724e0ea1f84f648598304e with delete_type TABLET_DATA_TOMBSTONED (TS e9596bfb8d324b4da93a4288771ed177 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47880
I20240502 07:59:07.230545 29229 tablet_replica.cc:330] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177: stopping tablet replica
I20240502 07:59:07.230702 29229 raft_consensus.cc:2229] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:07.230894 29229 raft_consensus.cc:2258] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:07.231859 29229 ts_tablet_manager.cc:1875] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:07.233604 29229 ts_tablet_manager.cc:1888] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:07.233675 29229 log.cc:1196] T fc1ac05112724e0ea1f84f648598304e P e9596bfb8d324b4da93a4288771ed177: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/wals/fc1ac05112724e0ea1f84f648598304e
I20240502 07:59:07.234241 25342 catalog_manager.cc:4830] TS e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865): tablet fc1ac05112724e0ea1f84f648598304e (table rebalance_test_table_5 [id=a5520c28ad46406f9fdf6855d5dfa37b]) successfully deleted
I20240502 07:59:07.238054 29666 tablet_copy_client.cc:523] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:07.238360 28885 tablet_copy_client.cc:523] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:07.239590 29666 tablet_bootstrap.cc:492] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap starting.
I20240502 07:59:07.240013 28885 tablet_bootstrap.cc:492] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42: Bootstrap starting.
I20240502 07:59:07.282507 25758 consensus_queue.cc:237] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.284328 25495 raft_consensus.cc:1261] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:07.284581 25890 raft_consensus.cc:1261] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:07.284783 28960 consensus_queue.cc:1035] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:07.285090 28464 consensus_queue.cc:1035] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:59:07.285732 25693 consensus_peers.cc:485] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f21d6014b91c45df96325017b57f19e3. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.287976 29617 tablet_bootstrap.cc:492] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:07.290074 29617 tablet_bootstrap.cc:492] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42: Bootstrap complete.
I20240502 07:59:07.290315 29617 ts_tablet_manager.cc:1367] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42: Time spent bootstrapping tablet: real 0.204s user 0.031s sys 0.000s
I20240502 07:59:07.290774 29617 raft_consensus.cc:348] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:07.290899 29617 raft_consensus.cc:729] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6746058199446968d2496df55cdee42, State: Initialized, Role: LEARNER
I20240502 07:59:07.290987 29617 consensus_queue.cc:260] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:07.291253 29617 ts_tablet_manager.cc:1398] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:07.291688 25911 tablet_copy_service.cc:342] P e9596bfb8d324b4da93a4288771ed177: Request end of tablet copy session c6746058199446968d2496df55cdee42-d6929d2560404acb9bb53f54a181f958 received from {username='slave'} at 127.18.13.135:50611
I20240502 07:59:07.291771 25911 tablet_copy_service.cc:433] P e9596bfb8d324b4da93a4288771ed177: ending tablet copy session c6746058199446968d2496df55cdee42-d6929d2560404acb9bb53f54a181f958 on tablet d6929d2560404acb9bb53f54a181f958 with peer c6746058199446968d2496df55cdee42
I20240502 07:59:07.293247 25497 consensus_queue.cc:237] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.294082 28960 raft_consensus.cc:2932] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.295328 25890 raft_consensus.cc:2932] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.297188 25352 catalog_manager.cc:5484] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.297531 27539 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 07:59:07.297806 25431 consensus_peers.cc:485] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1fa713b432740d98bcab918d251b54d. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.297919 28374 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:07.301126 25497 raft_consensus.cc:2932] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.301545 28885 tablet_bootstrap.cc:492] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:07.301823 28885 tablet_bootstrap.cc:492] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42: Bootstrap complete.
I20240502 07:59:07.301983 28885 ts_tablet_manager.cc:1367] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42: Time spent bootstrapping tablet: real 0.062s user 0.023s sys 0.000s
I20240502 07:59:07.302407 28885 raft_consensus.cc:348] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:07.302656 28885 raft_consensus.cc:729] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6746058199446968d2496df55cdee42, State: Initialized, Role: LEARNER
I20240502 07:59:07.302768 28885 consensus_queue.cc:260] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:07.303014 28885 ts_tablet_manager.cc:1398] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 07:59:07.303056 27274 raft_consensus.cc:1261] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:07.303745 25517 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session c6746058199446968d2496df55cdee42-16a130f60be74f568cc53a70692a6838 received from {username='slave'} at 127.18.13.135:42665
I20240502 07:59:07.303853 25517 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session c6746058199446968d2496df55cdee42-16a130f60be74f568cc53a70692a6838 on tablet 16a130f60be74f568cc53a70692a6838 with peer c6746058199446968d2496df55cdee42
I20240502 07:59:07.307770 29108 consensus_queue.cc:1035] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:07.312055 25497 consensus_queue.cc:237] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:07.313823 27671 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:07.314059 25890 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:07.314680 27539 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:07.315625 28374 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:07.315845 28375 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:07.316221 28675 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:07.316699 27539 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.317178 29177 raft_consensus.cc:1050] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177: attempting to promote NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 to VOTER
I20240502 07:59:07.317560 29177 consensus_queue.cc:237] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:07.318976 29108 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:07.319281 25626 raft_consensus.cc:1261] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:07.319370 25497 raft_consensus.cc:1261] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:07.319753 26712 consensus_queue.cc:1035] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:07.320103 26712 consensus_queue.cc:1035] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:07.320402 27671 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:07.320492 25349 catalog_manager.cc:5484] T a1fa713b432740d98bcab918d251b54d P c6746058199446968d2496df55cdee42 reported cstate change: config changed from index 9 to 10, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.322085 27405 raft_consensus.cc:1261] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:07.323948 25497 consensus_queue.cc:237] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } }
I20240502 07:59:07.323879 25348 catalog_manager.cc:5484] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:07.325053 29352 consensus_queue.cc:1035] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:07.325235 28515 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.325719 25758 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:07.326153 29109 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.326947 27405 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:07.327349 29109 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.327477 25626 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:07.328089 29109 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.330404 27539 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:07.331045 25497 consensus_queue.cc:237] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } }
I20240502 07:59:07.331552 27274 raft_consensus.cc:2932] T a1fa713b432740d98bcab918d251b54d P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.332669 27671 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:07.334170 27539 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:07.334602 28375 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:07.334721 28675 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:07.337491 29109 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } } }
W20240502 07:59:07.338069 25434 consensus_peers.cc:485] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): Couldn't send request to peer 0dfa00c061c748b18772d5f0566944b5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.339777 25626 consensus_queue.cc:237] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.340116 25343 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ee25634bf534443b90fdbdcebdbcb02e with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:59:07.340541 25497 consensus_queue.cc:237] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:07.341154 27671 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } } }
I20240502 07:59:07.342100 25759 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:07.342545 27670 raft_consensus.cc:1261] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:07.342514 29255 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:07.343632 27670 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:07.343924 25348 catalog_manager.cc:5484] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 9 to 10, NON_VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.344117 25626 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:07.344323 25889 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:07.344776 28374 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.344787 29255 consensus_queue.cc:1035] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:07.346004 29255 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
W20240502 07:59:07.346431 25562 consensus_peers.cc:485] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ad22d2e464d4defab6bd0061e6058f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.347019 28374 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.348793 28374 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:07.349514 25759 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.352361 25626 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:07.352530 25888 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:07.352501 25349 catalog_manager.cc:5484] T 1ad22d2e464d4defab6bd0061e6058f8 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 8 to 9, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.353029 25497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5a0605c8c64f4c6fb5e7315a118a0d7f"
dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43942
I20240502 07:59:07.352921 25890 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:07.353183 25497 raft_consensus.cc:595] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:07.353824 28675 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:07.354274 27671 raft_consensus.cc:2932] T 1ad22d2e464d4defab6bd0061e6058f8 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.354588 28375 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.354902 28375 raft_consensus.cc:979] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70: : Instructing follower 14d8fd1fc3a04d9d967b8033df84f34a to start an election
I20240502 07:59:07.354982 28375 raft_consensus.cc:1067] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Signalling peer 14d8fd1fc3a04d9d967b8033df84f34a to start an election
I20240502 07:59:07.356542 25353 catalog_manager.cc:5484] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.357012 27671 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:07.357715 25758 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5a0605c8c64f4c6fb5e7315a118a0d7f"
dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
from {username='slave'} at 127.18.13.129:39323
I20240502 07:59:07.357896 25758 raft_consensus.cc:482] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:07.358000 25758 raft_consensus.cc:3037] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:07.359059 25758 raft_consensus.cc:504] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } }
I20240502 07:59:07.359519 25758 leader_election.cc:288] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 2 election: Requested vote from peers c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377), 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037)
I20240502 07:59:07.359300 27539 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } } }
I20240502 07:59:07.360051 27405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a0605c8c64f4c6fb5e7315a118a0d7f" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "a17bbff6a71b4667b14f5dd12bca3057"
I20240502 07:59:07.362500 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a0605c8c64f4c6fb5e7315a118a0d7f" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
I20240502 07:59:07.362636 25497 raft_consensus.cc:3032] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:07.362684 25497 raft_consensus.cc:729] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Running, Role: LEADER
I20240502 07:59:07.362879 25497 consensus_queue.cc:260] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } }
I20240502 07:59:07.363082 25497 raft_consensus.cc:3037] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:07.364131 25497 raft_consensus.cc:2454] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 2.
I20240502 07:59:07.365752 25626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a0605c8c64f4c6fb5e7315a118a0d7f" candidate_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:59:07.365881 25626 raft_consensus.cc:3037] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:07.366912 25626 raft_consensus.cc:2454] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d8fd1fc3a04d9d967b8033df84f34a in term 2.
I20240502 07:59:07.367334 25890 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:07.367513 25696 leader_election.cc:302] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, 14d8fd1fc3a04d9d967b8033df84f34a, c3be11f0fe304e0f8e008dc1e4c41f70; no voters: a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:07.367692 28880 raft_consensus.cc:2781] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:07.367812 28880 raft_consensus.cc:686] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 2 LEADER]: Becoming Leader. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Running, Role: LEADER
I20240502 07:59:07.367959 28880 consensus_queue.cc:237] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } }
I20240502 07:59:07.368837 25890 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } } }
I20240502 07:59:07.370142 25353 catalog_manager.cc:5484] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: term changed from 1 to 2, leader changed from c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) to 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131), now has a pending config: VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131), VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129), VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130), VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134). New cstate: current_term: 2 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } }
I20240502 07:59:07.370370 25497 consensus_queue.cc:237] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.370625 25758 consensus_queue.cc:237] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:07.372359 29666 tablet_bootstrap.cc:492] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:07.372730 29666 tablet_bootstrap.cc:492] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap complete.
I20240502 07:59:07.373111 29666 ts_tablet_manager.cc:1367] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38: Time spent bootstrapping tablet: real 0.134s user 0.026s sys 0.007s
W20240502 07:59:07.373196 25431 consensus_peers.cc:485] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8904fa47fc274c14acd291b4c91964be. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.373379 27671 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:07.373785 29666 raft_consensus.cc:348] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:07.373827 29109 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:07.374001 29666 raft_consensus.cc:729] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85fbc6c98f0643469c92215a5cfb2e38, State: Initialized, Role: LEARNER
I20240502 07:59:07.374166 29666 consensus_queue.cc:260] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:07.374537 29666 ts_tablet_manager.cc:1398] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:07.375021 25517 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-eb0c33c3e4014eb3b931d64adac60e1a received from {username='slave'} at 127.18.13.136:58689
I20240502 07:59:07.375094 25517 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-eb0c33c3e4014eb3b931d64adac60e1a on tablet eb0c33c3e4014eb3b931d64adac60e1a with peer 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:07.376048 25693 consensus_peers.cc:485] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f21d6014b91c45df96325017b57f19e3. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.376833 25888 raft_consensus.cc:1261] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:07.377326 28464 consensus_queue.cc:1035] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:07.377933 25626 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:07.378598 28971 ts_tablet_manager.cc:902] T c95cd9642c1c46cabb23f6c62c6ad4f5 P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:59:07.378747 29109 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:07.379117 28971 tablet_copy_client.cc:308] T c95cd9642c1c46cabb23f6c62c6ad4f5 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.132:45865
I20240502 07:59:07.379163 29320 raft_consensus.cc:2932] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:07.379721 25888 raft_consensus.cc:2932] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
W20240502 07:59:07.379971 25696 consensus_peers.cc:485] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f21d6014b91c45df96325017b57f19e3. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.380353 25890 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:07.381204 25497 raft_consensus.cc:1261] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:07.381203 25911 tablet_copy_service.cc:140] P e9596bfb8d324b4da93a4288771ed177: Received BeginTabletCopySession request for tablet c95cd9642c1c46cabb23f6c62c6ad4f5 from peer a17bbff6a71b4667b14f5dd12bca3057 ({username='slave'} at 127.18.13.134:49677)
I20240502 07:59:07.381287 25911 tablet_copy_service.cc:161] P e9596bfb8d324b4da93a4288771ed177: Beginning new tablet copy session on tablet c95cd9642c1c46cabb23f6c62c6ad4f5 from peer a17bbff6a71b4667b14f5dd12bca3057 at {username='slave'} at 127.18.13.134:49677: session id = a17bbff6a71b4667b14f5dd12bca3057-c95cd9642c1c46cabb23f6c62c6ad4f5
I20240502 07:59:07.381553 29109 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:07.382308 25911 tablet_copy_source_session.cc:205] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:07.383324 29109 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.383845 28960 consensus_queue.cc:1035] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:07.384099 25348 catalog_manager.cc:5484] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.384353 29352 raft_consensus.cc:2932] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:07.385733 25758 consensus_queue.cc:237] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } }
I20240502 07:59:07.387091 27539 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:07.387200 25343 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8904fa47fc274c14acd291b4c91964be with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:59:07.387733 25890 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.387881 27405 raft_consensus.cc:2932] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:07.388392 29320 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:07.390049 25495 raft_consensus.cc:2932] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:07.390465 25348 catalog_manager.cc:5484] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057 reported cstate change: config changed from index 4 to 5, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:07.391472 25497 raft_consensus.cc:2932] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:07.392035 25626 raft_consensus.cc:2932] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:07.393509 25629 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.393926 27671 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.394820 28971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95cd9642c1c46cabb23f6c62c6ad4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:07.396157 25348 catalog_manager.cc:5484] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 6 to 7, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.397053 28971 tablet_copy_client.cc:783] T c95cd9642c1c46cabb23f6c62c6ad4f5 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:07.397217 28971 tablet_copy_client.cc:647] T c95cd9642c1c46cabb23f6c62c6ad4f5 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 1 WAL segments...
W20240502 07:59:07.398276 25696 consensus_peers.cc:485] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): Couldn't send request to peer 0dfa00c061c748b18772d5f0566944b5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.399297 28971 tablet_copy_client.cc:523] T c95cd9642c1c46cabb23f6c62c6ad4f5 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:07.399456 25497 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:07.399490 25495 consensus_queue.cc:237] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.400104 28464 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:07.400815 28971 tablet_bootstrap.cc:492] T c95cd9642c1c46cabb23f6c62c6ad4f5 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap starting.
I20240502 07:59:07.401455 27671 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:07.401693 25629 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:07.402166 28374 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:07.402710 27538 raft_consensus.cc:1203] T ff31ddd6b3d74a9aae6617afb56c40f5 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 07:59:07.403276 29109 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:07.404310 29109 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.406400 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8904fa47fc274c14acd291b4c91964be with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:07.406462 27671 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.406970 25629 raft_consensus.cc:2932] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
W20240502 07:59:07.409088 25431 consensus_peers.cc:485] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8904fa47fc274c14acd291b4c91964be. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.409734 25352 catalog_manager.cc:5484] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 7 to 8, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.410640 27589 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } } }
I20240502 07:59:07.411190 27539 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } } }
I20240502 07:59:07.414034 25348 catalog_manager.cc:5484] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 9 to 10, NON_VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.418143 25870 tablet_service.cc:1514] Processing DeleteTablet for tablet 8904fa47fc274c14acd291b4c91964be with delete_type TABLET_DATA_TOMBSTONED (TS e9596bfb8d324b4da93a4288771ed177 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:47880
I20240502 07:59:07.418426 29229 tablet_replica.cc:330] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177: stopping tablet replica
I20240502 07:59:07.418560 29229 raft_consensus.cc:2229] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:07.418759 29229 raft_consensus.cc:2258] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:07.420759 29229 ts_tablet_manager.cc:1875] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:07.420805 25495 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } } }
I20240502 07:59:07.421571 25888 consensus_queue.cc:237] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:07.423202 25497 raft_consensus.cc:1261] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:07.423429 25629 raft_consensus.cc:1261] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:07.423463 29229 ts_tablet_manager.cc:1888] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 07:59:07.423633 29229 log.cc:1196] T 8904fa47fc274c14acd291b4c91964be P e9596bfb8d324b4da93a4288771ed177: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/wals/8904fa47fc274c14acd291b4c91964be
I20240502 07:59:07.423987 27405 raft_consensus.cc:1261] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:07.423988 26712 consensus_queue.cc:1035] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.424330 26712 consensus_queue.cc:1035] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.424461 29168 consensus_queue.cc:1035] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.424638 25342 catalog_manager.cc:4830] TS e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865): tablet 8904fa47fc274c14acd291b4c91964be (table rebalance_test_table_4 [id=f72d1ad6e00b46cf99f485a9ef60b9b1]) successfully deleted
I20240502 07:59:07.426307 29177 raft_consensus.cc:2932] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:07.427090 25497 raft_consensus.cc:2932] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:07.429961 27405 raft_consensus.cc:2932] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:07.432559 25352 catalog_manager.cc:5484] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057 reported cstate change: config changed from index 5 to 6, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
W20240502 07:59:07.438086 25825 consensus_peers.cc:485] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbfd23aa383f413cbe4a5ccd8146d2b6. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.441533 28971 tablet_bootstrap.cc:492] T c95cd9642c1c46cabb23f6c62c6ad4f5 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:07.441834 28971 tablet_bootstrap.cc:492] T c95cd9642c1c46cabb23f6c62c6ad4f5 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap complete.
I20240502 07:59:07.442020 28971 ts_tablet_manager.cc:1367] T c95cd9642c1c46cabb23f6c62c6ad4f5 P a17bbff6a71b4667b14f5dd12bca3057: Time spent bootstrapping tablet: real 0.041s user 0.025s sys 0.000s
I20240502 07:59:07.442476 28971 raft_consensus.cc:348] T c95cd9642c1c46cabb23f6c62c6ad4f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.442595 28971 raft_consensus.cc:729] T c95cd9642c1c46cabb23f6c62c6ad4f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a17bbff6a71b4667b14f5dd12bca3057, State: Initialized, Role: LEARNER
I20240502 07:59:07.442682 28971 consensus_queue.cc:260] T c95cd9642c1c46cabb23f6c62c6ad4f5 P a17bbff6a71b4667b14f5dd12bca3057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.442935 28971 ts_tablet_manager.cc:1398] T c95cd9642c1c46cabb23f6c62c6ad4f5 P a17bbff6a71b4667b14f5dd12bca3057: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:07.443442 25911 tablet_copy_service.cc:342] P e9596bfb8d324b4da93a4288771ed177: Request end of tablet copy session a17bbff6a71b4667b14f5dd12bca3057-c95cd9642c1c46cabb23f6c62c6ad4f5 received from {username='slave'} at 127.18.13.134:49677
I20240502 07:59:07.443526 25911 tablet_copy_service.cc:433] P e9596bfb8d324b4da93a4288771ed177: ending tablet copy session a17bbff6a71b4667b14f5dd12bca3057-c95cd9642c1c46cabb23f6c62c6ad4f5 on tablet c95cd9642c1c46cabb23f6c62c6ad4f5 with peer a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:07.443735 25629 raft_consensus.cc:2932] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:07.446808 25758 consensus_queue.cc:237] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.448514 25626 raft_consensus.cc:1261] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:07.448766 27539 raft_consensus.cc:1261] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 07:59:07.448814 25693 consensus_peers.cc:485] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5958bb5c8fc4c1d91ae8f1e8b3861d0. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.449295 29320 consensus_queue.cc:1035] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:07.449772 29320 consensus_queue.cc:1035] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:07.451658 29320 raft_consensus.cc:2932] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.452512 25626 raft_consensus.cc:2932] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.454696 25348 catalog_manager.cc:5484] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 6 to 7, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.458949 25888 consensus_queue.cc:237] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:07.458917 27539 raft_consensus.cc:2932] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.460886 25495 raft_consensus.cc:1261] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:07.461355 25626 raft_consensus.cc:1261] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:07.462093 26712 consensus_queue.cc:1035] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:59:07.462296 25827 consensus_peers.cc:485] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 037fed57490949bbbafa11ebb44d4bb8. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.462442 26712 consensus_queue.cc:1035] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:07.467837 29168 raft_consensus.cc:2932] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:07.470993 27539 raft_consensus.cc:1203] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 07:59:07.471767 25348 catalog_manager.cc:5484] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.472951 25495 raft_consensus.cc:2932] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:07.478338 27274 raft_consensus.cc:1203] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:07.485018 25626 raft_consensus.cc:2932] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:07.508091 25758 consensus_queue.cc:237] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
W20240502 07:59:07.509727 25693 consensus_peers.cc:485] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03016ecfa11d4987a2e9d2368a66a5a2. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.509788 25495 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:07.510001 27539 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:07.510166 28464 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:07.510529 29321 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:07.512502 29321 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.513162 27539 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.515233 25352 catalog_manager.cc:5484] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 9 to 10, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.521078 27405 raft_consensus.cc:1203] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:07.522789 25497 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:07.532634 25497 consensus_queue.cc:237] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:07.534166 27671 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 07:59:07.534519 25433 consensus_peers.cc:485] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb0c33c3e4014eb3b931d64adac60e1a. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.534554 27274 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:07.534596 28374 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:07.535341 25626 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:07.535560 29108 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:07.536134 28374 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:07.537509 28374 raft_consensus.cc:1013] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: attempt to promote peer 85fbc6c98f0643469c92215a5cfb2e38: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 07:59:07.537741 28374 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.538537 27274 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.540498 25758 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "87d6266008784883891ca32c01363dbe"
dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51654
I20240502 07:59:07.540654 25758 raft_consensus.cc:595] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:07.540668 25626 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.541487 27671 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.542151 25352 catalog_manager.cc:5484] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 reported cstate change: config changed from index 7 to 8, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.543959 28675 raft_consensus.cc:1050] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 to VOTER
I20240502 07:59:07.544322 28675 consensus_queue.cc:237] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
W20240502 07:59:07.545974 25433 consensus_peers.cc:485] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb0c33c3e4014eb3b931d64adac60e1a. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.546279 27671 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:07.546707 28515 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:07.548518 25626 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:07.549218 27274 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:07.549584 28374 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:07.549701 28515 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:07.551932 28515 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.554136 27671 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.554536 25626 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.556614 27274 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.557585 25349 catalog_manager.cc:5484] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 8 to 9, 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.563891 25497 consensus_queue.cc:237] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }
I20240502 07:59:07.565568 27671 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:07.566727 27274 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:07.566943 28375 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:07.567148 28374 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:07.567531 25626 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:07.569082 28374 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:07.569371 28374 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:07.571342 27671 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:07.573189 27275 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:07.573321 25353 catalog_manager.cc:5484] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 9 to 10, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:07.573690 25626 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } }
I20240502 07:59:07.576197 27518 tablet_service.cc:1514] Processing DeleteTablet for tablet eb0c33c3e4014eb3b931d64adac60e1a with delete_type TABLET_DATA_TOMBSTONED (TS c6746058199446968d2496df55cdee42 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56612
I20240502 07:59:07.576876 27589 raft_consensus.cc:979] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a: : Instructing follower e9596bfb8d324b4da93a4288771ed177 to start an election
I20240502 07:59:07.576992 27589 raft_consensus.cc:1067] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Signalling peer e9596bfb8d324b4da93a4288771ed177 to start an election
W20240502 07:59:07.577142 25342 catalog_manager.cc:4797] TS c6746058199446968d2496df55cdee42 (127.18.13.135:34615): delete failed for tablet eb0c33c3e4014eb3b931d64adac60e1a because the tablet was not found. No further retry: Not found: Tablet not found: eb0c33c3e4014eb3b931d64adac60e1a
I20240502 07:59:07.577611 25888 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "87d6266008784883891ca32c01363dbe"
dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
from {username='slave'} at 127.18.13.131:35187
I20240502 07:59:07.577780 25888 raft_consensus.cc:482] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:07.577862 25888 raft_consensus.cc:3037] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:07.578912 25888 raft_consensus.cc:504] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:07.579408 25888 leader_election.cc:288] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 2 election: Requested vote from peers 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205), c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097), a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037)
I20240502 07:59:07.579663 25758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d6266008784883891ca32c01363dbe" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "14d8fd1fc3a04d9d967b8033df84f34a"
I20240502 07:59:07.579779 25758 raft_consensus.cc:3032] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:07.579865 25758 raft_consensus.cc:729] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 14d8fd1fc3a04d9d967b8033df84f34a, State: Running, Role: LEADER
I20240502 07:59:07.579850 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb0c33c3e4014eb3b931d64adac60e1a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:07.579994 27405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d6266008784883891ca32c01363dbe" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "a17bbff6a71b4667b14f5dd12bca3057"
I20240502 07:59:07.580032 25758 consensus_queue.cc:260] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:07.580133 27405 raft_consensus.cc:3037] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:07.580217 25758 raft_consensus.cc:3037] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:07.581246 27405 raft_consensus.cc:2454] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 2.
I20240502 07:59:07.581372 25758 raft_consensus.cc:2454] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 2.
I20240502 07:59:07.582419 25824 leader_election.cc:302] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 14d8fd1fc3a04d9d967b8033df84f34a, a17bbff6a71b4667b14f5dd12bca3057, e9596bfb8d324b4da93a4288771ed177; no voters:
I20240502 07:59:07.582598 26712 raft_consensus.cc:2781] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:07.582679 26712 raft_consensus.cc:686] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 2 LEADER]: Becoming Leader. State: Replica: e9596bfb8d324b4da93a4288771ed177, State: Running, Role: LEADER
I20240502 07:59:07.582814 26712 consensus_queue.cc:237] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:07.585237 27274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87d6266008784883891ca32c01363dbe" candidate_uuid: "e9596bfb8d324b4da93a4288771ed177" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "c5f3003b08fb4214b116c0be8dd62851"
I20240502 07:59:07.585450 27274 raft_consensus.cc:3037] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:07.585680 25352 catalog_manager.cc:5484] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 reported cstate change: term changed from 1 to 2, leader changed from 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) to e9596bfb8d324b4da93a4288771ed177 (127.18.13.132). New cstate: current_term: 2 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.586524 27274 raft_consensus.cc:2454] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9596bfb8d324b4da93a4288771ed177 in term 2.
I20240502 07:59:07.616231 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:07.618995 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:07.623339 25477 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:07.644055 27255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.646261 27518 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.648413 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.650574 25870 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:59:07.671284 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.685818 25497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff31ddd6b3d74a9aae6617afb56c40f5"
dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44052
I20240502 07:59:07.685966 25497 raft_consensus.cc:595] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:07.718864 28971 ts_tablet_manager.cc:902] T 1ad22d2e464d4defab6bd0061e6058f8 P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825)
I20240502 07:59:07.719343 28971 tablet_copy_client.cc:308] T 1ad22d2e464d4defab6bd0061e6058f8 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:38825
I20240502 07:59:07.732759 25649 tablet_copy_service.cc:140] P 0dfa00c061c748b18772d5f0566944b5: Received BeginTabletCopySession request for tablet 1ad22d2e464d4defab6bd0061e6058f8 from peer a17bbff6a71b4667b14f5dd12bca3057 ({username='slave'} at 127.18.13.134:59601)
I20240502 07:59:07.732887 25649 tablet_copy_service.cc:161] P 0dfa00c061c748b18772d5f0566944b5: Beginning new tablet copy session on tablet 1ad22d2e464d4defab6bd0061e6058f8 from peer a17bbff6a71b4667b14f5dd12bca3057 at {username='slave'} at 127.18.13.134:59601: session id = a17bbff6a71b4667b14f5dd12bca3057-1ad22d2e464d4defab6bd0061e6058f8
I20240502 07:59:07.733952 25649 tablet_copy_source_session.cc:205] T 1ad22d2e464d4defab6bd0061e6058f8 P 0dfa00c061c748b18772d5f0566944b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:07.734925 28971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ad22d2e464d4defab6bd0061e6058f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:07.746311 25497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8904fa47fc274c14acd291b4c91964be"
dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44076
I20240502 07:59:07.746481 25497 raft_consensus.cc:595] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:07.754078 28971 tablet_copy_client.cc:783] T 1ad22d2e464d4defab6bd0061e6058f8 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:07.754258 28971 tablet_copy_client.cc:647] T 1ad22d2e464d4defab6bd0061e6058f8 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:07.755880 28971 tablet_copy_client.cc:523] T 1ad22d2e464d4defab6bd0061e6058f8 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:07.757797 28971 tablet_bootstrap.cc:492] T 1ad22d2e464d4defab6bd0061e6058f8 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap starting.
I20240502 07:59:07.759608 27405 raft_consensus.cc:1203] T c95cd9642c1c46cabb23f6c62c6ad4f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:07.760000 27539 raft_consensus.cc:1203] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:07.764513 25626 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 2 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 07:59:07.765179 28960 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.781847 25758 consensus_queue.cc:237] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } }
I20240502 07:59:07.783342 25888 raft_consensus.cc:1261] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:07.783728 27589 consensus_queue.cc:1035] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:07.784126 27671 raft_consensus.cc:1261] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:07.784512 27589 consensus_queue.cc:1035] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 07:59:07.784725 25696 consensus_peers.cc:485] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): Couldn't send request to peer 0dfa00c061c748b18772d5f0566944b5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.785583 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:07.786571 28880 raft_consensus.cc:2932] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } } }
I20240502 07:59:07.788573 27671 raft_consensus.cc:2932] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } } }
I20240502 07:59:07.789727 25353 catalog_manager.cc:5484] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 6 to 7, NON_VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.790983 28675 raft_consensus.cc:1050] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER c6746058199446968d2496df55cdee42 to VOTER
W20240502 07:59:07.791059 28675 raft_consensus.cc:1053] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70: Unable to promote non-voter c6746058199446968d2496df55cdee42: Service unavailable: leader transfer in progress
I20240502 07:59:07.794620 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:07.799219 30097 ts_tablet_manager.cc:902] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:07.800364 27651 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.803275 30097 tablet_copy_client.cc:308] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:07.809221 25517 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet ee25634bf534443b90fdbdcebdbcb02e from peer 0dfa00c061c748b18772d5f0566944b5 ({username='slave'} at 127.18.13.130:33727)
I20240502 07:59:07.809345 25517 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet ee25634bf534443b90fdbdcebdbcb02e from peer 0dfa00c061c748b18772d5f0566944b5 at {username='slave'} at 127.18.13.130:33727: session id = 0dfa00c061c748b18772d5f0566944b5-ee25634bf534443b90fdbdcebdbcb02e
I20240502 07:59:07.810348 25517 tablet_copy_source_session.cc:205] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:07.812937 27387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.814925 27405 raft_consensus.cc:3037] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:07.816069 27405 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057 [term 2 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 07:59:07.816550 27589 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.818186 28880 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 2 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } }
I20240502 07:59:07.818531 27405 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057 [term 2 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } }
I20240502 07:59:07.819093 30097 ts_tablet_manager.cc:1875] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 07:59:07.821380 30097 ts_tablet_manager.cc:1888] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 07:59:07.821573 30097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee25634bf534443b90fdbdcebdbcb02e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:07.821689 25353 catalog_manager.cc:5484] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 5 to 6, no longer has a pending config: VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131), VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129), VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130), VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134). New cstate: current_term: 2 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } }
I20240502 07:59:07.822616 25497 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 2 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 07:59:07.825139 25626 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 2 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } }
I20240502 07:59:07.826494 27518 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.832886 27589 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:07.834304 25758 consensus_queue.cc:237] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:07.834645 25497 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 2 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } }
I20240502 07:59:07.838519 25626 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 2 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240502 07:59:07.838973 25696 consensus_peers.cc:485] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a0605c8c64f4c6fb5e7315a118a0d7f. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.839079 27589 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:07.840209 28375 raft_consensus.cc:979] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70: : Instructing follower a17bbff6a71b4667b14f5dd12bca3057 to start an election
I20240502 07:59:07.840305 28375 raft_consensus.cc:1067] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Signalling peer a17bbff6a71b4667b14f5dd12bca3057 to start an election
I20240502 07:59:07.840701 27407 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff31ddd6b3d74a9aae6617afb56c40f5"
dest_uuid: "a17bbff6a71b4667b14f5dd12bca3057"
from {username='slave'} at 127.18.13.129:52929
I20240502 07:59:07.840831 27405 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057 [term 2 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 07:59:07.840866 27407 raft_consensus.cc:482] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:07.840919 27407 raft_consensus.cc:3037] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:07.841951 27407 raft_consensus.cc:504] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:07.843065 27407 leader_election.cc:288] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [CANDIDATE]: Term 2 election: Requested vote from peers c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377), e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:59:07.844739 25496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff31ddd6b3d74a9aae6617afb56c40f5" candidate_uuid: "a17bbff6a71b4667b14f5dd12bca3057" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
I20240502 07:59:07.844863 25496 raft_consensus.cc:3032] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:07.844907 25496 raft_consensus.cc:729] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Running, Role: LEADER
I20240502 07:59:07.845083 25890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff31ddd6b3d74a9aae6617afb56c40f5" candidate_uuid: "a17bbff6a71b4667b14f5dd12bca3057" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "e9596bfb8d324b4da93a4288771ed177"
I20240502 07:59:07.845043 25496 consensus_queue.cc:260] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:07.845196 25890 raft_consensus.cc:3037] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:07.845256 25496 raft_consensus.cc:3037] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:07.845680 27589 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:07.846170 25890 raft_consensus.cc:2454] T ff31ddd6b3d74a9aae6617afb56c40f5 P e9596bfb8d324b4da93a4288771ed177 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a17bbff6a71b4667b14f5dd12bca3057 in term 2.
I20240502 07:59:07.846170 25496 raft_consensus.cc:2454] T ff31ddd6b3d74a9aae6617afb56c40f5 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a17bbff6a71b4667b14f5dd12bca3057 in term 2.
I20240502 07:59:07.846405 25888 raft_consensus.cc:2932] T 71a67cb0f6b54bfb9eb44400fe21ce40 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } } }
I20240502 07:59:07.846941 30104 ts_tablet_manager.cc:902] T f21d6014b91c45df96325017b57f19e3 P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:07.847599 30104 tablet_copy_client.cc:308] T f21d6014b91c45df96325017b57f19e3 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:07.848464 27344 leader_election.cc:302] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a17bbff6a71b4667b14f5dd12bca3057, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:59:07.848506 25779 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet f21d6014b91c45df96325017b57f19e3 from peer a17bbff6a71b4667b14f5dd12bca3057 ({username='slave'} at 127.18.13.134:33979)
I20240502 07:59:07.848598 25779 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet f21d6014b91c45df96325017b57f19e3 from peer a17bbff6a71b4667b14f5dd12bca3057 at {username='slave'} at 127.18.13.134:33979: session id = a17bbff6a71b4667b14f5dd12bca3057-f21d6014b91c45df96325017b57f19e3
I20240502 07:59:07.848744 29347 raft_consensus.cc:2781] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:07.848821 29347 raft_consensus.cc:686] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [term 2 LEADER]: Becoming Leader. State: Replica: a17bbff6a71b4667b14f5dd12bca3057, State: Running, Role: LEADER
I20240502 07:59:07.849092 29347 consensus_queue.cc:237] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:07.849130 25496 consensus_queue.cc:237] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } }
I20240502 07:59:07.849500 25779 tablet_copy_source_session.cc:205] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:07.850665 27671 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:59:07.850865 27539 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:59:07.851254 28375 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:07.851881 28960 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:07.852890 29108 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:07.852886 27407 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P a17bbff6a71b4667b14f5dd12bca3057 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:07.854136 29108 raft_consensus.cc:979] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70: : Instructing follower 85fbc6c98f0643469c92215a5cfb2e38 to start an election
I20240502 07:59:07.854182 25341 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5a0605c8c64f4c6fb5e7315a118a0d7f with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:59:07.854223 29108 raft_consensus.cc:1067] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Signalling peer 85fbc6c98f0643469c92215a5cfb2e38 to start an election
I20240502 07:59:07.854991 27670 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8904fa47fc274c14acd291b4c91964be"
dest_uuid: "85fbc6c98f0643469c92215a5cfb2e38"
from {username='slave'} at 127.18.13.129:57955
I20240502 07:59:07.855145 27670 raft_consensus.cc:482] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:07.855257 27670 raft_consensus.cc:3037] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:07.855149 25626 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:07.856279 27670 raft_consensus.cc:504] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
W20240502 07:59:07.856984 25434 consensus_peers.cc:485] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): Couldn't send request to peer 0dfa00c061c748b18772d5f0566944b5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.857043 30097 tablet_copy_client.cc:783] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:07.857220 30097 tablet_copy_client.cc:647] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:07.857940 30104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21d6014b91c45df96325017b57f19e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:07.857811 28675 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } } }
I20240502 07:59:07.858645 25353 catalog_manager.cc:5484] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 6 to 8, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 2 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.860605 27255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:07.862761 25350 catalog_manager.cc:5484] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 10 to 11, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.864742 27671 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } } }
I20240502 07:59:07.865425 27670 leader_election.cc:288] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [CANDIDATE]: Term 2 election: Requested vote from peers 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825), c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:07.866981 25626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8904fa47fc274c14acd291b4c91964be" candidate_uuid: "85fbc6c98f0643469c92215a5cfb2e38" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "0dfa00c061c748b18772d5f0566944b5"
I20240502 07:59:07.867113 25626 raft_consensus.cc:3037] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:07.868029 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ee25634bf534443b90fdbdcebdbcb02e with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:07.868117 25626 raft_consensus.cc:2454] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85fbc6c98f0643469c92215a5cfb2e38 in term 2.
I20240502 07:59:07.868470 27607 leader_election.cc:302] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfa00c061c748b18772d5f0566944b5, 85fbc6c98f0643469c92215a5cfb2e38; no voters:
I20240502 07:59:07.869158 25356 catalog_manager.cc:5484] T ff31ddd6b3d74a9aae6617afb56c40f5 P a17bbff6a71b4667b14f5dd12bca3057 reported cstate change: term changed from 1 to 2, leader changed from c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) to a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134). New cstate: current_term: 2 leader_uuid: "a17bbff6a71b4667b14f5dd12bca3057" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.869448 25498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8904fa47fc274c14acd291b4c91964be" candidate_uuid: "85fbc6c98f0643469c92215a5cfb2e38" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
I20240502 07:59:07.869567 25498 raft_consensus.cc:3032] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:07.869609 25498 raft_consensus.cc:729] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Running, Role: LEADER
I20240502 07:59:07.869777 28846 raft_consensus.cc:2781] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:07.869763 25498 consensus_queue.cc:260] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.869959 28846 raft_consensus.cc:686] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [term 2 LEADER]: Becoming Leader. State: Replica: 85fbc6c98f0643469c92215a5cfb2e38, State: Running, Role: LEADER
I20240502 07:59:07.870002 25498 raft_consensus.cc:3037] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:07.870270 28846 consensus_queue.cc:237] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.870982 25498 raft_consensus.cc:2454] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85fbc6c98f0643469c92215a5cfb2e38 in term 2.
I20240502 07:59:07.871905 27539 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } } }
I20240502 07:59:07.872354 25495 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 2 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 07:59:07.873097 27589 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:07.874583 25870 tablet_service.cc:1514] Processing DeleteTablet for tablet ee25634bf534443b90fdbdcebdbcb02e with delete_type TABLET_DATA_TOMBSTONED (TS e9596bfb8d324b4da93a4288771ed177 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:47880
I20240502 07:59:07.875217 29229 tablet_replica.cc:330] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177: stopping tablet replica
I20240502 07:59:07.875375 29229 raft_consensus.cc:2229] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:07.875591 29229 raft_consensus.cc:2258] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:07.876485 29229 ts_tablet_manager.cc:1875] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:07.876982 25353 catalog_manager.cc:5484] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 reported cstate change: term changed from 1 to 2, leader changed from c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) to 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136). New cstate: current_term: 2 leader_uuid: "85fbc6c98f0643469c92215a5cfb2e38" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.877545 30097 tablet_copy_client.cc:523] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:07.878778 29229 ts_tablet_manager.cc:1888] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 07:59:07.878851 29229 log.cc:1196] T ee25634bf534443b90fdbdcebdbcb02e P e9596bfb8d324b4da93a4288771ed177: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/wals/ee25634bf534443b90fdbdcebdbcb02e
I20240502 07:59:07.880556 30097 tablet_bootstrap.cc:492] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:59:07.880972 25496 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:07.882174 25477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 07:59:07.883756 25342 catalog_manager.cc:4830] TS e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865): tablet ee25634bf534443b90fdbdcebdbcb02e (table rebalance_test_table_0 [id=616a1d4f85774bf6b08fc4da5a27702a]) successfully deleted
I20240502 07:59:07.885944 30104 tablet_copy_client.cc:783] T f21d6014b91c45df96325017b57f19e3 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:07.886226 30104 tablet_copy_client.cc:647] T f21d6014b91c45df96325017b57f19e3 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:07.886667 28971 tablet_bootstrap.cc:492] T 1ad22d2e464d4defab6bd0061e6058f8 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:07.886974 28971 tablet_bootstrap.cc:492] T 1ad22d2e464d4defab6bd0061e6058f8 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap complete.
I20240502 07:59:07.887240 28971 ts_tablet_manager.cc:1367] T 1ad22d2e464d4defab6bd0061e6058f8 P a17bbff6a71b4667b14f5dd12bca3057: Time spent bootstrapping tablet: real 0.129s user 0.028s sys 0.000s
I20240502 07:59:07.887809 28971 raft_consensus.cc:348] T 1ad22d2e464d4defab6bd0061e6058f8 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.888015 28971 raft_consensus.cc:729] T 1ad22d2e464d4defab6bd0061e6058f8 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a17bbff6a71b4667b14f5dd12bca3057, State: Initialized, Role: LEARNER
I20240502 07:59:07.888154 28971 consensus_queue.cc:260] T 1ad22d2e464d4defab6bd0061e6058f8 P a17bbff6a71b4667b14f5dd12bca3057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.888520 28971 ts_tablet_manager.cc:1398] T 1ad22d2e464d4defab6bd0061e6058f8 P a17bbff6a71b4667b14f5dd12bca3057: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:07.888993 25649 tablet_copy_service.cc:342] P 0dfa00c061c748b18772d5f0566944b5: Request end of tablet copy session a17bbff6a71b4667b14f5dd12bca3057-1ad22d2e464d4defab6bd0061e6058f8 received from {username='slave'} at 127.18.13.134:59601
I20240502 07:59:07.889066 25649 tablet_copy_service.cc:433] P 0dfa00c061c748b18772d5f0566944b5: ending tablet copy session a17bbff6a71b4667b14f5dd12bca3057-1ad22d2e464d4defab6bd0061e6058f8 on tablet 1ad22d2e464d4defab6bd0061e6058f8 with peer a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:07.892372 28971 ts_tablet_manager.cc:902] T a1fa713b432740d98bcab918d251b54d P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:07.892809 28971 tablet_copy_client.cc:308] T a1fa713b432740d98bcab918d251b54d P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
I20240502 07:59:07.893239 25517 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet a1fa713b432740d98bcab918d251b54d from peer a17bbff6a71b4667b14f5dd12bca3057 ({username='slave'} at 127.18.13.134:48333)
I20240502 07:59:07.893343 25517 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet a1fa713b432740d98bcab918d251b54d from peer a17bbff6a71b4667b14f5dd12bca3057 at {username='slave'} at 127.18.13.134:48333: session id = a17bbff6a71b4667b14f5dd12bca3057-a1fa713b432740d98bcab918d251b54d
I20240502 07:59:07.894191 25517 tablet_copy_source_session.cc:205] T a1fa713b432740d98bcab918d251b54d P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:07.895069 28971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1fa713b432740d98bcab918d251b54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:07.897711 25870 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:59:07.906436 28971 tablet_copy_client.cc:783] T a1fa713b432740d98bcab918d251b54d P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:07.906649 28971 tablet_copy_client.cc:647] T a1fa713b432740d98bcab918d251b54d P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:07.909715 28971 tablet_copy_client.cc:523] T a1fa713b432740d98bcab918d251b54d P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:07.912222 28971 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap starting.
I20240502 07:59:07.915719 30142 ts_tablet_manager.cc:902] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 85fbc6c98f0643469c92215a5cfb2e38: Initiating tablet copy from peer e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:59:07.915733 25495 consensus_queue.cc:237] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:07.916457 30142 tablet_copy_client.cc:308] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.132:45865
I20240502 07:59:07.917363 27274 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:59:07.917758 28675 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 07:59:07.918759 25433 consensus_peers.cc:485] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb0c33c3e4014eb3b931d64adac60e1a. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.919869 25626 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:59:07.918170 27671 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:59:07.920428 28675 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:07.920547 29108 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:07.921361 25888 consensus_queue.cc:237] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:07.922955 28675 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
W20240502 07:59:07.923856 25825 consensus_peers.cc:485] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbfd23aa383f413cbe4a5ccd8146d2b6. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.924418 27407 raft_consensus.cc:1261] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:07.924945 29177 consensus_queue.cc:1035] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:07.926247 25496 raft_consensus.cc:1261] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:07.926798 26712 consensus_queue.cc:1035] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:07.927024 26712 raft_consensus.cc:2932] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:07.928448 27671 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.928596 27275 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.928701 25496 raft_consensus.cc:2932] T dbfd23aa383f413cbe4a5ccd8146d2b6 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:07.931407 25353 catalog_manager.cc:5484] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 reported cstate change: config changed from index 10 to 11, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.932489 27407 raft_consensus.cc:2932] T dbfd23aa383f413cbe4a5ccd8146d2b6 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:07.935493 25342 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dbfd23aa383f413cbe4a5ccd8146d2b6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:07.936489 25626 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.938863 25349 catalog_manager.cc:5484] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 6 to 7, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.942229 25911 tablet_copy_service.cc:140] P e9596bfb8d324b4da93a4288771ed177: Received BeginTabletCopySession request for tablet dbfd23aa383f413cbe4a5ccd8146d2b6 from peer 85fbc6c98f0643469c92215a5cfb2e38 ({username='slave'} at 127.18.13.136:53775)
I20240502 07:59:07.942348 25911 tablet_copy_service.cc:161] P e9596bfb8d324b4da93a4288771ed177: Beginning new tablet copy session on tablet dbfd23aa383f413cbe4a5ccd8146d2b6 from peer 85fbc6c98f0643469c92215a5cfb2e38 at {username='slave'} at 127.18.13.136:53775: session id = 85fbc6c98f0643469c92215a5cfb2e38-dbfd23aa383f413cbe4a5ccd8146d2b6
I20240502 07:59:07.942690 25607 tablet_service.cc:1514] Processing DeleteTablet for tablet dbfd23aa383f413cbe4a5ccd8146d2b6 with delete_type TABLET_DATA_TOMBSTONED (TS 0dfa00c061c748b18772d5f0566944b5 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34854
I20240502 07:59:07.943365 25911 tablet_copy_source_session.cc:205] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:07.944429 25496 consensus_queue.cc:237] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:07.945295 29168 raft_consensus.cc:1050] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177: attempting to promote NON_VOTER c6746058199446968d2496df55cdee42 to VOTER
I20240502 07:59:07.945706 29168 consensus_queue.cc:237] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:07.946108 27671 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240502 07:59:07.946341 25433 consensus_peers.cc:485] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb0c33c3e4014eb3b931d64adac60e1a. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.946538 28374 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:59:07.947228 25496 raft_consensus.cc:1261] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:07.947851 27538 raft_consensus.cc:1261] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:07.947885 29352 consensus_queue.cc:1035] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:07.948307 29352 consensus_queue.cc:1035] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:07.948202 28374 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.950464 25343 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb0c33c3e4014eb3b931d64adac60e1a with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:07.951376 25353 catalog_manager.cc:5484] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 11 to 12, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:07.951936 30124 ts_tablet_manager.cc:902] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:07.952448 30124 tablet_copy_client.cc:308] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:07.952828 30154 ts_tablet_manager.cc:902] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851: Initiating tablet copy from peer e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:59:07.953336 30154 tablet_copy_client.cc:308] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.132:45865
I20240502 07:59:07.953758 25911 tablet_copy_service.cc:140] P e9596bfb8d324b4da93a4288771ed177: Received BeginTabletCopySession request for tablet 037fed57490949bbbafa11ebb44d4bb8 from peer c5f3003b08fb4214b116c0be8dd62851 ({username='slave'} at 127.18.13.133:39133)
I20240502 07:59:07.953807 25779 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet f21d6014b91c45df96325017b57f19e3 from peer c5f3003b08fb4214b116c0be8dd62851 ({username='slave'} at 127.18.13.133:39885)
I20240502 07:59:07.953856 25911 tablet_copy_service.cc:161] P e9596bfb8d324b4da93a4288771ed177: Beginning new tablet copy session on tablet 037fed57490949bbbafa11ebb44d4bb8 from peer c5f3003b08fb4214b116c0be8dd62851 at {username='slave'} at 127.18.13.133:39133: session id = c5f3003b08fb4214b116c0be8dd62851-037fed57490949bbbafa11ebb44d4bb8
I20240502 07:59:07.953900 25779 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet f21d6014b91c45df96325017b57f19e3 from peer c5f3003b08fb4214b116c0be8dd62851 at {username='slave'} at 127.18.13.133:39885: session id = c5f3003b08fb4214b116c0be8dd62851-f21d6014b91c45df96325017b57f19e3
I20240502 07:59:07.954746 25779 tablet_copy_source_session.cc:205] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:07.954746 25911 tablet_copy_source_session.cc:205] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:07.956132 27671 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.956684 30154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037fed57490949bbbafa11ebb44d4bb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:07.958726 30124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f21d6014b91c45df96325017b57f19e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:07.958782 27274 raft_consensus.cc:1261] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 07:59:07.959205 28515 consensus_queue.cc:1035] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:59:07.960742 25607 tablet_service.cc:1514] Processing DeleteTablet for tablet eb0c33c3e4014eb3b931d64adac60e1a with delete_type TABLET_DATA_TOMBSTONED (TS 0dfa00c061c748b18772d5f0566944b5 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:34854
I20240502 07:59:07.961839 27274 raft_consensus.cc:2932] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:07.962045 27407 raft_consensus.cc:1261] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:07.962522 26712 consensus_queue.cc:1035] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:07.962880 30142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbfd23aa383f413cbe4a5ccd8146d2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:07.961822 30124 tablet_copy_client.cc:783] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:07.963990 30155 tablet_replica.cc:330] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5: stopping tablet replica
I20240502 07:59:07.964180 30155 raft_consensus.cc:2229] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:07.964427 30155 raft_consensus.cc:2258] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:07.966420 30142 tablet_copy_client.cc:783] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:07.966596 30142 tablet_copy_client.cc:647] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:07.967296 30097 tablet_bootstrap.cc:492] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:07.967368 25888 consensus_queue.cc:237] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:07.967626 30097 tablet_bootstrap.cc:492] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: Bootstrap complete.
I20240502 07:59:07.967953 30097 ts_tablet_manager.cc:1367] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.087s user 0.025s sys 0.005s
I20240502 07:59:07.968902 27405 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 07:59:07.968928 25758 raft_consensus.cc:1261] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 07:59:07.969393 26712 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 07:59:07.969553 25827 consensus_peers.cc:485] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:07.969728 26712 consensus_queue.cc:1035] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:59:07.970830 30142 tablet_copy_client.cc:523] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:07.971369 30097 raft_consensus.cc:348] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } }
I20240502 07:59:07.971975 30097 raft_consensus.cc:729] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: LEARNER
I20240502 07:59:07.972172 30097 consensus_queue.cc:260] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } }
I20240502 07:59:07.972991 29169 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:07.973625 30097 ts_tablet_manager.cc:1398] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240502 07:59:07.974301 25517 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session 0dfa00c061c748b18772d5f0566944b5-ee25634bf534443b90fdbdcebdbcb02e received from {username='slave'} at 127.18.13.130:33727
I20240502 07:59:07.974377 25517 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session 0dfa00c061c748b18772d5f0566944b5-ee25634bf534443b90fdbdcebdbcb02e on tablet ee25634bf534443b90fdbdcebdbcb02e with peer 0dfa00c061c748b18772d5f0566944b5
I20240502 07:59:07.975535 27407 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:07.971652 30155 ts_tablet_manager.cc:1875] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:07.976423 30142 tablet_bootstrap.cc:492] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap starting.
I20240502 07:59:07.977545 28971 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:07.977468 25348 catalog_manager.cc:5484] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 7 to 8, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:07.977792 28971 tablet_bootstrap.cc:492] T a1fa713b432740d98bcab918d251b54d P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap complete.
I20240502 07:59:07.977944 28971 ts_tablet_manager.cc:1367] T a1fa713b432740d98bcab918d251b54d P a17bbff6a71b4667b14f5dd12bca3057: Time spent bootstrapping tablet: real 0.066s user 0.026s sys 0.003s
I20240502 07:59:07.978368 30155 ts_tablet_manager.cc:1888] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 07:59:07.978428 30155 log.cc:1196] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 0dfa00c061c748b18772d5f0566944b5: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/dbfd23aa383f413cbe4a5ccd8146d2b6
I20240502 07:59:07.978379 28971 raft_consensus.cc:348] T a1fa713b432740d98bcab918d251b54d P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.978479 28971 raft_consensus.cc:729] T a1fa713b432740d98bcab918d251b54d P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a17bbff6a71b4667b14f5dd12bca3057, State: Initialized, Role: LEARNER
I20240502 07:59:07.978367 27538 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:07.978566 28971 consensus_queue.cc:260] T a1fa713b432740d98bcab918d251b54d P a17bbff6a71b4667b14f5dd12bca3057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:07.978844 28971 ts_tablet_manager.cc:1398] T a1fa713b432740d98bcab918d251b54d P a17bbff6a71b4667b14f5dd12bca3057: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:07.979363 25517 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session a17bbff6a71b4667b14f5dd12bca3057-a1fa713b432740d98bcab918d251b54d received from {username='slave'} at 127.18.13.134:48333
I20240502 07:59:07.979454 25517 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session a17bbff6a71b4667b14f5dd12bca3057-a1fa713b432740d98bcab918d251b54d on tablet a1fa713b432740d98bcab918d251b54d with peer a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:07.980229 25343 catalog_manager.cc:4830] TS 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): tablet dbfd23aa383f413cbe4a5ccd8146d2b6 (table rebalance_test_table_7 [id=89a9dd84e8b5432a81ef7767804381d3]) successfully deleted
I20240502 07:59:07.980499 30155 tablet_replica.cc:330] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5: stopping tablet replica
I20240502 07:59:07.980607 30155 raft_consensus.cc:2229] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:07.980837 30155 raft_consensus.cc:2258] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:07.982736 29168 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:07.983105 25758 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:07.985280 30124 tablet_copy_client.cc:647] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:07.985168 27405 raft_consensus.cc:2932] T 0aa60608089c4f50bcee339d01815a65 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:07.985970 28971 ts_tablet_manager.cc:902] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
F20240502 07:59:07.986129 28971 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0aa60608089c4f50bcee339d01815a65.queue_time_us, tablet-download-pool-0aa60608089c4f50bcee339d01815a65.queue_time_us) (tablet-download-pool-0aa60608089c4f50bcee339d01815a65.run_cpu_time_us, tablet-download-pool-0aa60608089c4f50bcee339d01815a65.run_cpu_time_us) (tablet-download-pool-0aa60608089c4f50bcee339d01815a65.run_wall_time_us, tablet-download-pool-0aa60608089c4f50bcee339d01815a65.run_wall_time_us) (tablet-download-pool-1ad22d2e464d4defab6bd0061e6058f8.queue_time_us, tablet-download-pool-1ad22d2e464d4defab6bd0061e6058f8.queue_time_us) (tablet-download-pool-1ad22d2e464d4defab6bd0061e6058f8.run_cpu_time_us, tablet-download-pool-1ad22d2e464d4defab6bd0061e6058f8.run_cpu_time_us) (tablet-download-pool-1ad22d2e464d4defab6bd0061e6058f8.run_wall_time_us, tablet-download-pool-1ad22d2e464d4defab6bd0061e6058f8.run_wall_time_us) (tablet-download-pool-5a0605c8c64f4c6fb5e7315a118a0d7f.queue_time_us, tablet-download-pool-5a0605c8c64f4c6fb5e7315a118a0d7f.queue_time_us) (tablet-download-pool-5a0605c8c64f4c6fb5e7315a118a0d7f.run_cpu_time_us, tablet-download-pool-5a0605c8c64f4c6fb5e7315a118a0d7f.run_cpu_time_us) (tablet-download-pool-5a0605c8c64f4c6fb5e7315a118a0d7f.run_wall_time_us, tablet-download-pool-5a0605c8c64f4c6fb5e7315a118a0d7f.run_wall_time_us) (tablet-download-pool-843d1c1d6384476983749a78f65a5c74.queue_time_us, tablet-download-pool-843d1c1d6384476983749a78f65a5c74.queue_time_us) (tablet-download-pool-843d1c1d6384476983749a78f65a5c74.run_cpu_time_us, tablet-download-pool-843d1c1d6384476983749a78f65a5c74.run_cpu_time_us) (tablet-download-pool-843d1c1d6384476983749a78f65a5c74.run_wall_time_us, tablet-download-pool-843d1c1d6384476983749a78f65a5c74.run_wall_time_us) (tablet-download-pool-87d6266008784883891ca32c01363dbe.queue_time_us, tablet-download-pool-87d6266008784883891ca32c01363dbe.queue_time_us) (tablet-download-pool-87d6266008784883891ca32c01363dbe.run_cpu_time_us, tablet-download-pool-87d6266008784883891ca32c01363dbe.run_cpu_time_us) (tablet-download-pool-87d6266008784883891ca32c01363dbe.run_wall_time_us, tablet-download-pool-87d6266008784883891ca32c01363dbe.run_wall_time_us) (tablet-download-pool-9722dae4e4414084bdd8066b41bcc47a.queue_time_us, tablet-download-pool-9722dae4e4414084bdd8066b41bcc47a.queue_time_us) (tablet-download-pool-9722dae4e4414084bdd8066b41bcc47a.run_cpu_time_us, tablet-download-pool-9722dae4e4414084bdd8066b41bcc47a.run_cpu_time_us) (tablet-download-pool-9722dae4e4414084bdd8066b41bcc47a.run_wall_time_us, tablet-download-pool-9722dae4e4414084bdd8066b41bcc47a.run_wall_time_us) (tablet-download-pool-a1fa713b432740d98bcab918d251b54d.queue_time_us, tablet-download-pool-a1fa713b432740d98bcab918d251b54d.queue_time_us) (tablet-download-pool-a1fa713b432740d98bcab918d251b54d.run_cpu_time_us, tablet-download-pool-a1fa713b432740d98bcab918d251b54d.run_cpu_time_us) (tablet-download-pool-a1fa713b432740d98bcab918d251b54d.run_wall_time_us, tablet-download-pool-a1fa713b432740d98bcab918d251b54d.run_wall_time_us) (tablet-download-pool-c95cd9642c1c46cabb23f6c62c6ad4f5.queue_time_us, tablet-download-pool-c95cd9642c1c46cabb23f6c62c6ad4f5.queue_time_us) (tablet-download-pool-c95cd9642c1c46cabb23f6c62c6ad4f5.run_cpu_time_us, tablet-download-pool-c95cd9642c1c46cabb23f6c62c6ad4f5.run_cpu_time_us) (tablet-download-pool-c95cd9642c1c46cabb23f6c62c6ad4f5.run_wall_time_us, tablet-download-pool-c95cd9642c1c46cabb23f6c62c6ad4f5.run_wall_time_us) (tablet-download-pool-d6929d2560404acb9bb53f54a181f958.queue_time_us, tablet-download-pool-d6929d2560404acb9bb53f54a181f958.queue_time_us) (tablet-download-pool-d6929d2560404acb9bb53f54a181f958.run_cpu_time_us, tablet-download-pool-d6929d2560404acb9bb53f54a181f958.run_cpu_time_us) (tablet-download-pool-d6929d2560404acb9bb53f54a181f958.run_wall_time_us, tablet-download-pool-d6929d2560404acb9bb53f54a181f958.run_wall_time_us) (tablet-download-pool-d8e50f3d37d842868900a4cff5bf3601.queue_time_us, tablet-download-pool-d8e50f3d37d842868900a4cff5bf3601.queue_time_us) (tablet-download-pool-d8e50f3d37d842868900a4cff5bf3601.run_cpu_time_us, tablet-download-pool-d8e50f3d37d842868900a4cff5bf3601.run_cpu_time_us) (tablet-download-pool-d8e50f3d37d842868900a4cff5bf3601.run_wall_time_us, tablet-download-pool-d8e50f3d37d842868900a4cff5bf3601.run_wall_time_us) (tablet-download-pool-dbfd23aa383f413cbe4a5ccd8146d2b6.queue_time_us, tablet-download-pool-dbfd23aa383f413cbe4a5ccd8146d2b6.queue_time_us) (tablet-download-pool-dbfd23aa383f413cbe4a5ccd8146d2b6.run_cpu_time_us, tablet-download-pool-dbfd23aa383f413cbe4a5ccd8146d2b6.run_cpu_time_us) (tablet-download-pool-dbfd23aa383f413cbe4a5ccd8146d2b6.run_wall_time_us, tablet-download-pool-dbfd23aa383f413cbe4a5ccd8146d2b6.run_wall_time_us) (tablet-download-pool-e5958bb5c8fc4c1d91ae8f1e8b3861d0.queue_time_us, tablet-download-pool-e5958bb5c8fc4c1d91ae8f1e8b3861d0.queue_time_us) (tablet-download-pool-f21d6014b91c45df96325017b57f19e3.queue_time_us, tablet-download-pool-f21d6014b91c45df96325017b57f19e3.queue_time_us) (tablet-download-pool-f21d6014b91c45df96325017b57f19e3.run_cpu_time_us, tablet-download-pool-f21d6014b91c45df96325017b57f19e3.run_cpu_time_us) (tablet-download-pool-f21d6014b91c45df96325017b57f19e3.run_wall_time_us, tablet-download-pool-f21d6014b91c45df96325017b57f19e3.run_wall_time_us) (tablet-download-pool-fd68090fdadf45faa0e3b2936b30047e.queue_time_us, tablet-download-pool-fd68090fdadf45faa0e3b2936b30047e.queue_time_us) (tablet-download-pool-fd68090fdadf45faa0e3b2936b30047e.run_cpu_time_us, tablet-download-pool-fd68090fdadf45faa0e3b2936b30047e.run_cpu_time_us) (tablet-download-pool-fd68090fdadf45faa0e3b2936b30047e.run_wall_time_us, tablet-download-pool-fd68090fdadf45faa0e3b2936b30047e.run_wall_time_us) (tablet-download-pool-ff31ddd6b3d74a9aae6617afb56c40f5.queue_time_us, tablet-download-pool-ff31ddd6b3d74a9aae6617afb56c40f5.queue_time_us) (tablet-download-pool-ff31ddd6b3d74a9aae6617afb56c40f5.run_cpu_time_us, tablet-download-pool-ff31ddd6b3d74a9aae6617afb56c40f5.run_cpu_time_us) (tablet-download-pool-ff31ddd6b3d74a9aae6617afb56c40f5.run_wall_time_us, tablet-download-pool-ff31ddd6b3d74a9aae6617afb56c40f5.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240502 07:59:07.986522 25349 catalog_manager.cc:5484] T 0aa60608089c4f50bcee339d01815a65 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 11 to 12, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
*** Check failure stack trace: ***
I20240502 07:59:07.986877 30154 tablet_copy_client.cc:783] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:07.987057 30154 tablet_copy_client.cc:647] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 1 WAL segments...
@ 0x7f932b2efdcd google::LogMessage::Fail() at ??:0
I20240502 07:59:07.993499 30097 ts_tablet_manager.cc:902] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
@ 0x7f932b2f3b93 google::LogMessage::SendToLog() at ??:0
I20240502 07:59:07.994035 30097 tablet_copy_client.cc:308] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:07.994553 25779 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet 9722dae4e4414084bdd8066b41bcc47a from peer 0dfa00c061c748b18772d5f0566944b5 ({username='slave'} at 127.18.13.130:34585)
I20240502 07:59:07.994674 25779 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet 9722dae4e4414084bdd8066b41bcc47a from peer 0dfa00c061c748b18772d5f0566944b5 at {username='slave'} at 127.18.13.130:34585: session id = 0dfa00c061c748b18772d5f0566944b5-9722dae4e4414084bdd8066b41bcc47a
I20240502 07:59:07.995961 25779 tablet_copy_source_session.cc:205] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:07.996690 30097 ts_tablet_manager.cc:1875] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: Deleting tablet data with delete state TABLET_DATA_COPYING
@ 0x7f932b2ef7cc google::LogMessage::Flush() at ??:0
I20240502 07:59:07.997537 30154 tablet_copy_client.cc:523] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f932b2f0f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 07:59:07.999143 30154 tablet_bootstrap.cc:492] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap starting.
I20240502 07:59:08.001415 30097 ts_tablet_manager.cc:1888] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 07:59:08.001621 30097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9722dae4e4414084bdd8066b41bcc47a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:08.002527 30155 ts_tablet_manager.cc:1875] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
@ 0x7f932c759044 kudu::TraceMetrics::InternName() at ??:0
I20240502 07:59:08.003320 30097 tablet_copy_client.cc:783] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:08.003583 30097 tablet_copy_client.cc:647] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:08.005010 30097 tablet_copy_client.cc:523] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f932c73df98 kudu::ThreadPool::ThreadPool() at ??:0
I20240502 07:59:08.007381 30155 ts_tablet_manager.cc:1888] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 07:59:08.007457 30155 log.cc:1196] T eb0c33c3e4014eb3b931d64adac60e1a P 0dfa00c061c748b18772d5f0566944b5: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/eb0c33c3e4014eb3b931d64adac60e1a
I20240502 07:59:08.007539 30097 tablet_bootstrap.cc:492] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:59:08.008004 25343 catalog_manager.cc:4830] TS 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): tablet eb0c33c3e4014eb3b931d64adac60e1a (table rebalance_test_table_6 [id=67841c5c645f4e86970122328fd1238e]) successfully deleted
@ 0x7f932c73befc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f932f2ee77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f932f2f7a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f932f37876f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f932f38c4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f932f377226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240502 07:59:08.030597 30103 ts_tablet_manager.cc:902] T 03016ecfa11d4987a2e9d2368a66a5a2 P a17bbff6a71b4667b14f5dd12bca3057: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:08.031036 30104 tablet_copy_client.cc:523] T f21d6014b91c45df96325017b57f19e3 P a17bbff6a71b4667b14f5dd12bca3057: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:08.032588 30104 tablet_bootstrap.cc:492] T f21d6014b91c45df96325017b57f19e3 P a17bbff6a71b4667b14f5dd12bca3057: Bootstrap starting.
I20240502 07:59:08.040344 27274 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851 [term 2 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 07:59:08.040882 29352 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
@ 0x7f932f386c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240502 07:59:08.058758 30154 tablet_bootstrap.cc:492] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:08.059090 30154 tablet_bootstrap.cc:492] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap complete.
I20240502 07:59:08.059264 30154 ts_tablet_manager.cc:1367] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851: Time spent bootstrapping tablet: real 0.060s user 0.029s sys 0.000s
I20240502 07:59:08.059757 30154 raft_consensus.cc:348] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:08.059914 30154 raft_consensus.cc:729] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5f3003b08fb4214b116c0be8dd62851, State: Initialized, Role: LEARNER
I20240502 07:59:08.060042 30154 consensus_queue.cc:260] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:08.060357 30154 ts_tablet_manager.cc:1398] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:08.060824 25911 tablet_copy_service.cc:342] P e9596bfb8d324b4da93a4288771ed177: Request end of tablet copy session c5f3003b08fb4214b116c0be8dd62851-037fed57490949bbbafa11ebb44d4bb8 received from {username='slave'} at 127.18.13.133:39133
I20240502 07:59:08.060914 25911 tablet_copy_service.cc:433] P e9596bfb8d324b4da93a4288771ed177: ending tablet copy session c5f3003b08fb4214b116c0be8dd62851-037fed57490949bbbafa11ebb44d4bb8 on tablet 037fed57490949bbbafa11ebb44d4bb8 with peer c5f3003b08fb4214b116c0be8dd62851
@ 0x55a42a45a75c std::function<>::operator()() at ??:0
I20240502 07:59:08.066190 30097 tablet_bootstrap.cc:492] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:08.066519 30097 tablet_bootstrap.cc:492] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: Bootstrap complete.
I20240502 07:59:08.066718 30097 ts_tablet_manager.cc:1367] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.059s user 0.032s sys 0.000s
I20240502 07:59:08.067301 30097 raft_consensus.cc:348] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } }
I20240502 07:59:08.067462 30097 raft_consensus.cc:729] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: LEARNER
I20240502 07:59:08.067622 30097 consensus_queue.cc:260] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } }
I20240502 07:59:08.070461 30124 tablet_copy_client.cc:523] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:08.072098 30124 tablet_bootstrap.cc:492] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap starting.
I20240502 07:59:08.073223 25758 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 2 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 07:59:08.074499 29352 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:08.076329 25496 raft_consensus.cc:2932] T d6929d2560404acb9bb53f54a181f958 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:08.084923 27405 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057 [term 2 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 07:59:08.085366 29177 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:08.087157 25759 consensus_queue.cc:237] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:08.089598 27538 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 07:59:08.090070 25693 consensus_peers.cc:485] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03016ecfa11d4987a2e9d2368a66a5a2. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.090672 25496 raft_consensus.cc:1261] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:59:08.090742 29320 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:08.091089 29320 consensus_queue.cc:1035] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 07:59:08.093525 25694 consensus_peers.cc:485] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03016ecfa11d4987a2e9d2368a66a5a2. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.093848 27589 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:08.094233 25496 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:08.094440 27538 raft_consensus.cc:2932] T 03016ecfa11d4987a2e9d2368a66a5a2 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:08.096905 25353 catalog_manager.cc:5484] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 10 to 11, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:08.098172 30097 ts_tablet_manager.cc:1398] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.031s user 0.003s sys 0.000s
I20240502 07:59:08.100733 25779 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session 0dfa00c061c748b18772d5f0566944b5-9722dae4e4414084bdd8066b41bcc47a received from {username='slave'} at 127.18.13.130:34585
I20240502 07:59:08.100809 25779 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session 0dfa00c061c748b18772d5f0566944b5-9722dae4e4414084bdd8066b41bcc47a on tablet 9722dae4e4414084bdd8066b41bcc47a with peer 0dfa00c061c748b18772d5f0566944b5
I20240502 07:59:08.101253 29177 raft_consensus.cc:1050] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177: attempting to promote NON_VOTER c5f3003b08fb4214b116c0be8dd62851 to VOTER
I20240502 07:59:08.101759 29177 consensus_queue.cc:237] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
@ 0x7f932c740e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 07:59:08.103905 27407 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:08.104638 29352 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
@ 0x7f932c7416ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240502 07:59:08.106246 25626 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:08.106798 27274 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:08.107205 29169 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:08.107462 29169 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:08.109452 25759 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:08.109457 29352 raft_consensus.cc:1013] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: attempt to promote peer a17bbff6a71b4667b14f5dd12bca3057: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 07:59:08.109849 29169 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
@ 0x7f932c74303b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55a42a45a75c std::function<>::operator()() at ??:0
I20240502 07:59:08.116876 29168 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:08.119047 27407 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:08.121197 27274 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:08.121860 25352 catalog_manager.cc:5484] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20240502 07:59:08.122927 25626 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:08.126564 25759 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } }
I20240502 07:59:08.131222 29169 raft_consensus.cc:1050] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177: attempting to promote NON_VOTER a17bbff6a71b4667b14f5dd12bca3057 to VOTER
I20240502 07:59:08.131832 29169 consensus_queue.cc:237] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
W20240502 07:59:08.133569 25342 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 843d1c1d6384476983749a78f65a5c74 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:08.133996 25759 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:08.134366 29177 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:08.134476 27274 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:08.134928 29177 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:08.135695 25626 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:08.136190 29177 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:08.136945 27407 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 LEARNER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:08.137331 29177 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:08.142465 25888 consensus_queue.cc:237] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:08.143802 27405 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P a17bbff6a71b4667b14f5dd12bca3057 [term 2 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:59:08.143854 27275 raft_consensus.cc:1261] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851 [term 2 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
@ 0x7f932c730807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f932cf6e6db start_thread at ??:0
I20240502 07:59:08.149256 29169 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:08.151265 29177 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:08.151671 29177 consensus_queue.cc:1035] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:08.151585 27405 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P a17bbff6a71b4667b14f5dd12bca3057 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:08.153187 27275 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:08.154834 25626 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:08.156051 25353 catalog_manager.cc:5484] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851 reported cstate change: config changed from index 6 to 7, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:08.158241 25759 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:08.161010 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:08.162029 29177 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:08.162900 27274 raft_consensus.cc:2932] T 87d6266008784883891ca32c01363dbe P c5f3003b08fb4214b116c0be8dd62851 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:08.163967 25342 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 87d6266008784883891ca32c01363dbe with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:08.164551 25890 consensus_queue.cc:237] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:08.166453 25759 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
@ 0x7f932a4a671f clone at ??:0
I20240502 07:59:08.167647 26712 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:08.168481 25348 catalog_manager.cc:5484] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 8 to 10, VOTER 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131) evicted. New cstate: current_term: 2 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:08.172619 25738 tablet_service.cc:1514] Processing DeleteTablet for tablet 87d6266008784883891ca32c01363dbe with delete_type TABLET_DATA_TOMBSTONED (TS 14d8fd1fc3a04d9d967b8033df84f34a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55602
I20240502 07:59:08.173031 27651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:08.176144 27274 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:08.176579 29168 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:08.183501 25870 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:59:08.191812 25477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 07:59:08.193593 25342 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 843d1c1d6384476983749a78f65a5c74 with cas_config_opid_index 6: aborting the task: latest config opid_index 7; task opid_index 6
I20240502 07:59:08.195621 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:08.196753 27255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:08.197350 27518 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:08.202559 25759 consensus_queue.cc:237] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
W20240502 07:59:08.204124 25695 consensus_peers.cc:485] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer c6746058199446968d2496df55cdee42 (127.18.13.135:34615): Couldn't send request to peer c6746058199446968d2496df55cdee42. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99c3ddedadbf4c0e8858085b66022342. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.204560 25626 raft_consensus.cc:1261] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:08.204942 27671 raft_consensus.cc:1261] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:08.205310 28464 consensus_queue.cc:1035] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:08.205988 29320 consensus_queue.cc:1035] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:08.210922 26712 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:08.212848 25342 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 843d1c1d6384476983749a78f65a5c74 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:08.213610 30097 ts_tablet_manager.cc:902] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:08.213475 25758 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:08.214138 30097 tablet_copy_client.cc:308] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:08.214221 27274 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:08.214555 25779 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet 71a67cb0f6b54bfb9eb44400fe21ce40 from peer 0dfa00c061c748b18772d5f0566944b5 ({username='slave'} at 127.18.13.130:34585)
I20240502 07:59:08.214659 25779 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet 71a67cb0f6b54bfb9eb44400fe21ce40 from peer 0dfa00c061c748b18772d5f0566944b5 at {username='slave'} at 127.18.13.130:34585: session id = 0dfa00c061c748b18772d5f0566944b5-71a67cb0f6b54bfb9eb44400fe21ce40
I20240502 07:59:08.215559 25779 tablet_copy_source_session.cc:205] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:08.216534 30097 ts_tablet_manager.cc:1875] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 07:59:08.218708 30097 ts_tablet_manager.cc:1888] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 07:59:08.218880 30097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a67cb0f6b54bfb9eb44400fe21ce40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:08.224016 30097 tablet_copy_client.cc:783] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:08.299962 25496 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P c3be11f0fe304e0f8e008dc1e4c41f70 [term 2 FOLLOWER]: Refusing update from remote peer 85fbc6c98f0643469c92215a5cfb2e38: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 07:59:08.235095 30263 tablet_replica.cc:330] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a: stopping tablet replica
I20240502 07:59:08.305917 28515 raft_consensus.cc:1050] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER c6746058199446968d2496df55cdee42 to VOTER
I20240502 07:59:08.270471 29320 raft_consensus.cc:2932] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:08.408563 30263 raft_consensus.cc:2229] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:08.408820 30263 raft_consensus.cc:2258] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:08.408879 28515 consensus_queue.cc:237] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:08.409590 30263 ts_tablet_manager.cc:1875] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:08.410706 25758 raft_consensus.cc:1261] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:08.411229 28675 consensus_queue.cc:1035] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:08.411892 30263 ts_tablet_manager.cc:1888] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 07:59:08.414284 25477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 07:59:08.472893 25477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 07:59:08.224215 30097 tablet_copy_client.cc:647] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:08.271026 25626 raft_consensus.cc:2932] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:08.411983 30263 log.cc:1196] T 87d6266008784883891ca32c01363dbe P 14d8fd1fc3a04d9d967b8033df84f34a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/87d6266008784883891ca32c01363dbe
I20240502 07:59:08.414865 25738 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:08.338689 25870 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:59:08.467649 25871 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:59:08.230901 30124 tablet_bootstrap.cc:492] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:08.478978 30124 tablet_bootstrap.cc:492] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap complete.
I20240502 07:59:08.384634 25629 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEARNER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } } }
I20240502 07:59:08.479187 30124 ts_tablet_manager.cc:1367] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851: Time spent bootstrapping tablet: real 0.407s user 0.030s sys 0.000s
I20240502 07:59:08.479771 30124 raft_consensus.cc:348] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:08.479974 30124 raft_consensus.cc:729] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5f3003b08fb4214b116c0be8dd62851, State: Initialized, Role: LEARNER
I20240502 07:59:08.457445 25739 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:08.412125 25627 raft_consensus.cc:1261] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:08.481582 29109 consensus_queue.cc:1035] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:08.490938 30097 tablet_copy_client.cc:523] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:08.518067 25739 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:08.530957 25871 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:59:08.493181 30097 tablet_bootstrap.cc:492] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap starting.
I20240502 07:59:08.505000 25607 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:08.505617 25609 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:08.511369 25608 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:08.539752 25627 raft_consensus.cc:2932] T 16a130f60be74f568cc53a70692a6838 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
W20240502 07:59:08.551189 25693 connection.cc:535] client connection to 127.18.13.134:43037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 07:59:08.567481 25693 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 07:59:08.551169 25562 connection.cc:535] client connection to 127.18.13.134:43037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 07:59:08.522388 25477 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 07:59:08.568003 25562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 07:59:08.480070 30124 consensus_queue.cc:260] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:08.568513 30124 ts_tablet_manager.cc:1398] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851: Time spent starting tablet: real 0.089s user 0.000s sys 0.002s
W20240502 07:59:08.550762 25824 connection.cc:535] client connection to 127.18.13.134:43037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 07:59:08.568835 25824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 07:59:08.538722 28675 raft_consensus.cc:2932] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:08.330576 27255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:08.400648 27274 raft_consensus.cc:1203] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 07:59:08.551287 25431 connection.cc:535] client connection to 127.18.13.134:43037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 07:59:08.570979 25431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 07:59:08.424762 30377 ts_tablet_manager.cc:902] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: Initiating tablet copy from peer e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865)
I20240502 07:59:08.571828 30377 tablet_copy_client.cc:308] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.132:45865
I20240502 07:59:08.292176 30154 ts_tablet_manager.cc:902] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c5f3003b08fb4214b116c0be8dd62851: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
W20240502 07:59:08.628260 25431 consensus_peers.cc:485] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.598579 25628 raft_consensus.cc:1203] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 07:59:08.631281 30097 tablet_bootstrap.cc:492] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:08.639356 30097 tablet_bootstrap.cc:492] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: Bootstrap complete.
I20240502 07:59:08.639662 30097 ts_tablet_manager.cc:1367] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: Time spent bootstrapping tablet: real 0.147s user 0.029s sys 0.000s
I20240502 07:59:08.640223 30097 raft_consensus.cc:348] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } }
I20240502 07:59:08.640465 30097 raft_consensus.cc:729] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0dfa00c061c748b18772d5f0566944b5, State: Initialized, Role: LEARNER
I20240502 07:59:08.640666 30097 consensus_queue.cc:260] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: true } }
I20240502 07:59:08.641083 30097 ts_tablet_manager.cc:1398] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:08.572762 25779 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session c5f3003b08fb4214b116c0be8dd62851-f21d6014b91c45df96325017b57f19e3 received from {username='slave'} at 127.18.13.133:39885
I20240502 07:59:08.643107 25779 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session c5f3003b08fb4214b116c0be8dd62851-f21d6014b91c45df96325017b57f19e3 on tablet f21d6014b91c45df96325017b57f19e3 with peer c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:08.216601 25352 catalog_manager.cc:5484] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 7 to 8, VOTER 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } }
I20240502 07:59:08.573314 25758 raft_consensus.cc:2932] T 16a130f60be74f568cc53a70692a6838 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:08.233832 30142 tablet_bootstrap.cc:492] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:08.644140 30142 tablet_bootstrap.cc:492] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap complete.
W20240502 07:59:08.617122 25693 consensus_peers.cc:462] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111)
W20240502 07:59:08.644240 25693 consensus_peers.cc:485] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.644342 30142 ts_tablet_manager.cc:1367] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent bootstrapping tablet: real 0.668s user 0.022s sys 0.008s
I20240502 07:59:08.644789 30142 raft_consensus.cc:348] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:08.644902 30142 raft_consensus.cc:729] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85fbc6c98f0643469c92215a5cfb2e38, State: Initialized, Role: LEARNER
I20240502 07:59:08.644996 30142 consensus_queue.cc:260] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 85fbc6c98f0643469c92215a5cfb2e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:08.645143 25780 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session 0dfa00c061c748b18772d5f0566944b5-71a67cb0f6b54bfb9eb44400fe21ce40 received from {username='slave'} at 127.18.13.130:34585
I20240502 07:59:08.645203 25780 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session 0dfa00c061c748b18772d5f0566944b5-71a67cb0f6b54bfb9eb44400fe21ce40 on tablet 71a67cb0f6b54bfb9eb44400fe21ce40 with peer 0dfa00c061c748b18772d5f0566944b5
I20240502 07:59:08.645316 30142 ts_tablet_manager.cc:1398] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:08.481356 25341 catalog_manager.cc:4830] TS 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205): tablet 87d6266008784883891ca32c01363dbe (table rebalance_test_table_1 [id=a5f087acc4d94b0f9694fb1cf95e7dca]) successfully deleted
W20240502 07:59:08.645601 25341 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.18.13.131:42205, user_credentials={real_user=slave}} blocked reactor thread for 164226us
I20240502 07:59:08.272415 27670 raft_consensus.cc:2932] T 99c3ddedadbf4c0e8858085b66022342 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:08.413180 30125 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 07:59:08.646889 25693 consensus_peers.cc:462] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111)
W20240502 07:59:08.551872 30148 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 07:59:08.404590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.13.134:43037 (local address 127.18.13.136:52973)
0502 07:59:08.404796 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0502 07:59:08.404814 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 07:59:08.404920 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 07:59:08.551303 (+146383us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":41}
I20240502 07:59:08.494733 25356 catalog_manager.cc:5484] T 99c3ddedadbf4c0e8858085b66022342 P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 6 to 7, NON_VOTER c6746058199446968d2496df55cdee42 (127.18.13.135) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } } }
I20240502 07:59:08.575381 25347 catalog_manager.cc:5484] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 4 to 5, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:08.572659 30154 tablet_copy_client.cc:308] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:08.575560 27255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:08.586313 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:08.650380 25780 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet 5a0605c8c64f4c6fb5e7315a118a0d7f from peer c5f3003b08fb4214b116c0be8dd62851 ({username='slave'} at 127.18.13.133:39885)
I20240502 07:59:08.650475 25780 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet 5a0605c8c64f4c6fb5e7315a118a0d7f from peer c5f3003b08fb4214b116c0be8dd62851 at {username='slave'} at 127.18.13.133:39885: session id = c5f3003b08fb4214b116c0be8dd62851-5a0605c8c64f4c6fb5e7315a118a0d7f
I20240502 07:59:08.651368 25780 tablet_copy_source_session.cc:205] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:08.605507 27721 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskeLuhys/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 07:59:02.367249 27729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:02.367540 27729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:02.372058 27729 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:59:03.475687 27729 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 07:59:03.989396 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7c7fb9171c9548a7ad2d45e295d780cb: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:03.989538 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:03.989634 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:03.989723 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c6746058199446968d2496df55cdee42
W20240502 07:59:03.989778 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 616a1d4f85774bf6b08fc4da5a27702a: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:03.989822 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f72d1ad6e00b46cf99f485a9ef60b9b1: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c6746058199446968d2496df55cdee42
W20240502 07:59:03.989913 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:03.989960 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:03.989995 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:03.990023 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:03.990054 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:03.990084 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 616a1d4f85774bf6b08fc4da5a27702a: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:03.990142 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7c7fb9171c9548a7ad2d45e295d780cb: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:03.990175 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c6746058199446968d2496df55cdee42
W20240502 07:59:03.990207 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8aaedf9499044a93ab9419f10e2342e2: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c6746058199446968d2496df55cdee42
W20240502 07:59:03.990245 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c6746058199446968d2496df55cdee42
W20240502 07:59:03.990293 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:03.990324 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:04.022722 27729 rebalancer_tool.cc:1074] tablet 056ba8d15d18409e946c905b8876097a: 'e9596bfb8d324b4da93a4288771ed177' -> '85fbc6c98f0643469c92215a5cfb2e38' move scheduled
I20240502 07:59:04.116295 27729 rebalancer_tool.cc:1074] tablet 9722dae4e4414084bdd8066b41bcc47a: '0dfa00c061c748b18772d5f0566944b5' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:04.169772 27729 rebalancer_tool.cc:1074] tablet 5a0605c8c64f4c6fb5e7315a118a0d7f: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:04.348659 27729 rebalancer_tool.cc:1074] tablet fd68090fdadf45faa0e3b2936b30047e: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'c5f3003b08fb4214b116c0be8dd62851' move scheduled
I20240502 07:59:04.396718 27729 rebalancer_tool.cc:1074] tablet c81a7795712e45b5a7fd97450a9f06a7: 'e9596bfb8d324b4da93a4288771ed177' -> '85fbc6c98f0643469c92215a5cfb2e38' move scheduled
I20240502 07:59:04.472731 27729 rebalancer_tool.cc:1074] tablet ee25634bf534443b90fdbdcebdbcb02e: '0dfa00c061c748b18772d5f0566944b5' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:04.566808 27729 rebalancer_tool.cc:1074] tablet 0aa60608089c4f50bcee339d01815a65: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'c5f3003b08fb4214b116c0be8dd62851' move scheduled
I20240502 07:59:04.606025 27729 rebalancer_tool.cc:1074] tablet 87d6266008784883891ca32c01363dbe: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:04.723955 27729 rebalancer_tool.cc:1074] tablet 21a8d5798c8943658113d67c88f4ad25: 'e9596bfb8d324b4da93a4288771ed177' -> '85fbc6c98f0643469c92215a5cfb2e38' move scheduled
I20240502 07:59:04.825300 27729 rebalancer_tool.cc:1074] tablet 03016ecfa11d4987a2e9d2368a66a5a2: '0dfa00c061c748b18772d5f0566944b5' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:04.892824 27729 rebalancer_tool.cc:1074] tablet 1ad22d2e464d4defab6bd0061e6058f8: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> '85fbc6c98f0643469c92215a5cfb2e38' move scheduled
I20240502 07:59:04.984210 27729 rebalancer_tool.cc:1074] tablet 273e203d4ed44c43859f08e9bec08a9d: 'e9596bfb8d324b4da93a4288771ed177' -> 'c5f3003b08fb4214b116c0be8dd62851' move scheduled
I20240502 07:59:05.046491 27729 rebalancer_tool.cc:1074] tablet d8e50f3d37d842868900a4cff5bf3601: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:05.093128 27729 rebalancer_tool.cc:1084] tablet a3bc416015dc40d88743215d2cf46230: '0dfa00c061c748b18772d5f0566944b5' -> 'c5f3003b08fb4214b116c0be8dd62851' move ignored: Illegal state: tablet a3bc416015dc40d88743215d2cf46230: replica c5f3003b08fb4214b116c0be8dd62851 is already present
I20240502 07:59:05.093237 27729 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:06.207764 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f72d1ad6e00b46cf99f485a9ef60b9b1: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c6746058199446968d2496df55cdee42
W20240502 07:59:06.207891 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c6746058199446968d2496df55cdee42
W20240502 07:59:06.207955 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:06.208005 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7c7fb9171c9548a7ad2d45e295d780cb: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:06.208029 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c6746058199446968d2496df55cdee42
W20240502 07:59:06.208053 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:06.208072 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:06.208098 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:06.208153 27729 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 616a1d4f85774bf6b08fc4da5a27702a: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c6746058199446968d2496df55cdee42
I20240502 07:59:06.365134 27729 rebalancer_tool.cc:1074] tablet ff31ddd6b3d74a9aae6617afb56c40f5: '14d8fd1fc3a04d9d967b8033df84f34a' -> '' move scheduled
I20240502 07:59:06.474438 27729 rebalancer_tool.cc:1074] tablet 843d1c1d6384476983749a78f65a5c74: '0dfa00c061c748b18772d5f0566944b5' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:06.521471 27729 rebalancer_tool.cc:1074] tablet d6929d2560404acb9bb53f54a181f958: 'e9596bfb8d324b4da93a4288771ed177' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:06.614037 27729 rebalancer_tool.cc:1074] tablet eb0c33c3e4014eb3b931d64adac60e1a: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> '85fbc6c98f0643469c92215a5cfb2e38' move scheduled
I20240502 07:59:06.680881 27729 rebalancer_tool.cc:1074] tablet 16a130f60be74f568cc53a70692a6838: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:06.831701 27729 rebalancer_tool.cc:1074] tablet c95cd9642c1c46cabb23f6c62c6ad4f5: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:07.115959 27729 rebalancer_tool.cc:1136] tablet c81a7795712e45b5a7fd97450a9f06a7: 'e9596bfb8d324b4da93a4288771ed177' -> '85fbc6c98f0643469c92215a5cfb2e38' move completed: OK
I20240502 07:59:07.422132 27729 rebalancer_tool.cc:1136] tablet 056ba8d15d18409e946c905b8876097a: 'e9596bfb8d324b4da93a4288771ed177' -> '85fbc6c98f0643469c92215a5cfb2e38' move completed: OK
I20240502 07:59:07.450109 27729 rebalancer_tool.cc:1123] tablet 9722dae4e4414084bdd8066b41bcc47a: '0dfa00c061c748b18772d5f0566944b5' -> 'c6746058199446968d2496df55cdee42' move is abandoned: Illegal state: 9722dae4e4414084bdd8066b41bcc47a: source replica 0dfa00c061c748b18772d5f0566944b5 does not have REPLACE attribute set
I20240502 07:59:07.472595 27729 rebalancer_tool.cc:1136] tablet 0aa60608089c4f50bcee339d01815a65: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'c5f3003b08fb4214b116c0be8dd62851' move completed: Incomplete: tablet 0aa60608089c4f50bcee339d01815a65, TS c3be11f0fe304e0f8e008dc1e4c41f70 -> TS c5f3003b08fb4214b116c0be8dd62851 move failed, target replica disappeared
I20240502 07:59:07.487157 27729 rebalancer_tool.cc:1136] tablet fd68090fdadf45faa0e3b2936b30047e: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'c5f3003b08fb4214b116c0be8dd62851' move completed: OK
I20240502 07:59:07.497088 27729 rebalancer_tool.cc:1123] tablet ee25634bf534443b90fdbdcebdbcb02e: '0dfa00c061c748b18772d5f0566944b5' -> 'c6746058199446968d2496df55cdee42' move is abandoned: Illegal state: ee25634bf534443b90fdbdcebdbcb02e: source replica 0dfa00c061c748b18772d5f0566944b5 does not have REPLACE attribute set
I20240502 07:59:07.503469 27729 rebalancer_tool.cc:1136] tablet 273e203d4ed44c43859f08e9bec08a9d: 'e9596bfb8d324b4da93a4288771ed177' -> 'c5f3003b08fb4214b116c0be8dd62851' move completed: OK
I20240502 07:59:07.511421 27729 rebalancer_tool.cc:1136] tablet ff31ddd6b3d74a9aae6617afb56c40f5: '14d8fd1fc3a04d9d967b8033df84f34a' -> '' move completed: OK
I20240502 07:59:07.584715 27729 rebalancer_tool.cc:1136] tablet 21a8d5798c8943658113d67c88f4ad25: 'e9596bfb8d324b4da93a4288771ed177' -> '85fbc6c98f0643469c92215a5cfb2e38' move completed: OK
I20240502 07:59:07.636267 27729 rebalancer_tool.cc:1136] tablet 03016ecfa11d4987a2e9d2368a66a5a2: '0dfa00c061c748b18772d5f0566944b5' -> 'c6746058199446968d2496df55cdee42' move completed: OK
I20240502 07:59:07.710214 27729 rebalancer_tool.cc:1136] tablet 1ad22d2e464d4defab6bd0061e6058f8: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> '85fbc6c98f0643469c92215a5cfb2e38' move completed: OK
I20240502 07:59:07.724579 27729 rebalancer_tool.cc:1136] tablet d8e50f3d37d842868900a4cff5bf3601: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'a17bbff6a71b4667b14f5dd12bca3057' move completed: OK
I20240502 07:59:07.754189 27729 rebalancer_tool.cc:1084] tablet f21d6014b91c45df96325017b57f19e3: 'e9596bfb8d324b4da93a4288771ed177' -> 'c5f3003b08fb4214b116c0be8dd62851' move ignored: Illegal state: tablet f21d6014b91c45df96325017b57f19e3: replica c5f3003b08fb4214b116c0be8dd62851 is already present
I20240502 07:59:07.754273 27729 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:08.548647 30254 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 07:59:08.145741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.13.134:43037 (local address 127.0.0.1:43612)
0502 07:59:08.155347 (+ 9606us) negotiation.cc:107] Waiting for socket to connect
0502 07:59:08.155360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0502 07:59:08.155474 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 07:59:08.159647 (+ 4173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 07:59:08.159653 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 07:59:08.159669 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0502 07:59:08.159953 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 07:59:08.159960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 07:59:08.164325 (+ 4365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0502 07:59:08.164328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0502 07:59:08.164771 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 07:59:08.164779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 07:59:08.164887 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0502 07:59:08.168094 (+ 3207us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0502 07:59:08.168096 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0502 07:59:08.168110 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0502 07:59:08.168137 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0502 07:59:08.547724 (+379587us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":9537,"thread_start_us":64,"threads_started":1}
W20240502 07:59:08.553552 30254 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 07:59:08.549476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.13.134:43037 (local address 127.0.0.1:43644)
0502 07:59:08.549648 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0502 07:59:08.549662 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0502 07:59:08.549741 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 07:59:08.553469 (+ 3728us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":99}
Illegal state: tablet server a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): unacceptable health status UNAVAILABLE
I20240502 07:59:08.573714 25911 tablet_copy_service.cc:140] P e9596bfb8d324b4da93a4288771ed177: Received BeginTabletCopySession request for tablet 0aa60608089c4f50bcee339d01815a65 from peer c5f3003b08fb4214b116c0be8dd62851 ({username='slave'} at 127.18.13.133:39133)
I20240502 07:59:08.652698 27651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:08.652695 25911 tablet_copy_service.cc:161] P e9596bfb8d324b4da93a4288771ed177: Beginning new tablet copy session on tablet 0aa60608089c4f50bcee339d01815a65 from peer c5f3003b08fb4214b116c0be8dd62851 at {username='slave'} at 127.18.13.133:39133: session id = c5f3003b08fb4214b116c0be8dd62851-0aa60608089c4f50bcee339d01815a65
I20240502 07:59:08.653690 25911 tablet_copy_source_session.cc:205] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 07:59:08.614630 25824 consensus_peers.cc:485] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:59:08.654021 25824 consensus_peers.cc:485] T 87d6266008784883891ca32c01363dbe P e9596bfb8d324b4da93a4288771ed177 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:59:08.654078 25824 consensus_peers.cc:485] T c95cd9642c1c46cabb23f6c62c6ad4f5 P e9596bfb8d324b4da93a4288771ed177 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:59:08.654119 25824 consensus_peers.cc:485] T dbfd23aa383f413cbe4a5ccd8146d2b6 P e9596bfb8d324b4da93a4288771ed177 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:59:08.654153 25824 consensus_peers.cc:485] T d6929d2560404acb9bb53f54a181f958 P e9596bfb8d324b4da93a4288771ed177 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:59:08.654186 25824 consensus_peers.cc:485] T 0aa60608089c4f50bcee339d01815a65 P e9596bfb8d324b4da93a4288771ed177 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.342648 27518 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:08.411866 27538 raft_consensus.cc:1261] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:08.475055 30400 ts_tablet_manager.cc:902] T eb0c33c3e4014eb3b931d64adac60e1a P c6746058199446968d2496df55cdee42: Initiating tablet copy from peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377)
I20240502 07:59:08.654664 30377 ts_tablet_manager.cc:1875] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 07:59:08.655174 30400 tablet_copy_client.cc:308] T eb0c33c3e4014eb3b931d64adac60e1a P c6746058199446968d2496df55cdee42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37377
W20240502 07:59:08.655239 27604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 07:59:08.614779 30425 ts_tablet_manager.cc:902] T 99c3ddedadbf4c0e8858085b66022342 P c6746058199446968d2496df55cdee42: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:08.655900 30425 tablet_copy_client.cc:308] T 99c3ddedadbf4c0e8858085b66022342 P c6746058199446968d2496df55cdee42: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:08.656733 25780 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet 99c3ddedadbf4c0e8858085b66022342 from peer c6746058199446968d2496df55cdee42 ({username='slave'} at 127.18.13.135:52983)
I20240502 07:59:08.656822 25780 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet 99c3ddedadbf4c0e8858085b66022342 from peer c6746058199446968d2496df55cdee42 at {username='slave'} at 127.18.13.135:52983: session id = c6746058199446968d2496df55cdee42-99c3ddedadbf4c0e8858085b66022342
I20240502 07:59:08.656865 30377 ts_tablet_manager.cc:1888] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.10
I20240502 07:59:08.657040 30377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa60608089c4f50bcee339d01815a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:08.657703 25780 tablet_copy_source_session.cc:205] T 99c3ddedadbf4c0e8858085b66022342 P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:08.657704 28515 consensus_queue.cc:1035] T 16a130f60be74f568cc53a70692a6838 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:08.658056 30377 tablet_copy_client.cc:783] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:08.658179 30377 tablet_copy_client.cc:647] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:08.658600 30425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c3ddedadbf4c0e8858085b66022342. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:08.660826 30425 tablet_copy_client.cc:783] T 99c3ddedadbf4c0e8858085b66022342 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:08.660970 30425 tablet_copy_client.cc:647] T 99c3ddedadbf4c0e8858085b66022342 P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:08.662036 25517 tablet_copy_service.cc:140] P c3be11f0fe304e0f8e008dc1e4c41f70: Received BeginTabletCopySession request for tablet eb0c33c3e4014eb3b931d64adac60e1a from peer c6746058199446968d2496df55cdee42 ({username='slave'} at 127.18.13.135:42665)
I20240502 07:59:08.662132 25517 tablet_copy_service.cc:161] P c3be11f0fe304e0f8e008dc1e4c41f70: Beginning new tablet copy session on tablet eb0c33c3e4014eb3b931d64adac60e1a from peer c6746058199446968d2496df55cdee42 at {username='slave'} at 127.18.13.135:42665: session id = c6746058199446968d2496df55cdee42-eb0c33c3e4014eb3b931d64adac60e1a
I20240502 07:59:08.662307 30425 tablet_copy_client.cc:523] T 99c3ddedadbf4c0e8858085b66022342 P c6746058199446968d2496df55cdee42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:08.662977 25517 tablet_copy_source_session.cc:205] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:08.663530 30425 tablet_bootstrap.cc:492] T 99c3ddedadbf4c0e8858085b66022342 P c6746058199446968d2496df55cdee42: Bootstrap starting.
I20240502 07:59:08.665827 30154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a0605c8c64f4c6fb5e7315a118a0d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:08.666177 25628 raft_consensus.cc:1261] T 8904fa47fc274c14acd291b4c91964be P 0dfa00c061c748b18772d5f0566944b5 [term 2 FOLLOWER]: Refusing update from remote peer 85fbc6c98f0643469c92215a5cfb2e38: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 07:59:08.666846 27538 raft_consensus.cc:2932] T 16a130f60be74f568cc53a70692a6838 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6746058199446968d2496df55cdee42 (127.18.13.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } }
I20240502 07:59:08.668956 30400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0c33c3e4014eb3b931d64adac60e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:08.669003 27651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:08.669163 30390 consensus_queue.cc:1035] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:08.669668 27518 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:08.670254 27518 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:08.671006 30400 tablet_copy_client.cc:783] T eb0c33c3e4014eb3b931d64adac60e1a P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:08.671140 30400 tablet_copy_client.cc:647] T eb0c33c3e4014eb3b931d64adac60e1a P c6746058199446968d2496df55cdee42: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:08.671156 30154 tablet_copy_client.cc:783] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:08.671310 30154 tablet_copy_client.cc:647] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:08.674829 30154 tablet_copy_client.cc:523] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:08.676403 30154 tablet_bootstrap.cc:492] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c5f3003b08fb4214b116c0be8dd62851: Bootstrap starting.
I20240502 07:59:08.676632 30377 tablet_copy_client.cc:523] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:08.677243 25608 tablet_service.cc:1514] Processing DeleteTablet for tablet 843d1c1d6384476983749a78f65a5c74 with delete_type TABLET_DATA_TOMBSTONED (TS 0dfa00c061c748b18772d5f0566944b5 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34854
I20240502 07:59:08.677794 25890 consensus_queue.cc:237] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:08.678730 30377 tablet_bootstrap.cc:492] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap starting.
I20240502 07:59:08.679438 27275 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:08.681372 26712 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:08.681481 27651 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:08.683351 25758 raft_consensus.cc:1261] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:08.684911 26712 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:08.686744 27275 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:08.693934 27275 raft_consensus.cc:1203] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:08.703126 30425 tablet_bootstrap.cc:492] T 99c3ddedadbf4c0e8858085b66022342 P c6746058199446968d2496df55cdee42: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:08.703460 30425 tablet_bootstrap.cc:492] T 99c3ddedadbf4c0e8858085b66022342 P c6746058199446968d2496df55cdee42: Bootstrap complete.
I20240502 07:59:08.703620 30425 ts_tablet_manager.cc:1367] T 99c3ddedadbf4c0e8858085b66022342 P c6746058199446968d2496df55cdee42: Time spent bootstrapping tablet: real 0.040s user 0.027s sys 0.000s
I20240502 07:59:08.704427 29169 consensus_queue.cc:1035] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:08.704545 25342 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 843d1c1d6384476983749a78f65a5c74 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:08.704861 30425 raft_consensus.cc:348] T 99c3ddedadbf4c0e8858085b66022342 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:08.705036 30425 raft_consensus.cc:729] T 99c3ddedadbf4c0e8858085b66022342 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6746058199446968d2496df55cdee42, State: Initialized, Role: LEARNER
I20240502 07:59:08.705145 30425 consensus_queue.cc:260] T 99c3ddedadbf4c0e8858085b66022342 P c6746058199446968d2496df55cdee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:08.706144 25352 catalog_manager.cc:5484] T 843d1c1d6384476983749a78f65a5c74 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 8 to 9, VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:08.706651 25758 raft_consensus.cc:2932] T 843d1c1d6384476983749a78f65a5c74 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:08.707202 30405 ts_tablet_manager.cc:902] T 03016ecfa11d4987a2e9d2368a66a5a2 P 85fbc6c98f0643469c92215a5cfb2e38: Initiating tablet copy from peer 14d8fd1fc3a04d9d967b8033df84f34a (127.18.13.131:42205)
I20240502 07:59:08.707760 30405 tablet_copy_client.cc:308] T 03016ecfa11d4987a2e9d2368a66a5a2 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:42205
I20240502 07:59:08.708279 25911 tablet_copy_service.cc:342] P e9596bfb8d324b4da93a4288771ed177: Request end of tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-dbfd23aa383f413cbe4a5ccd8146d2b6 received from {username='slave'} at 127.18.13.136:53775
I20240502 07:59:08.708355 25911 tablet_copy_service.cc:433] P e9596bfb8d324b4da93a4288771ed177: ending tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-dbfd23aa383f413cbe4a5ccd8146d2b6 on tablet dbfd23aa383f413cbe4a5ccd8146d2b6 with peer 85fbc6c98f0643469c92215a5cfb2e38
I20240502 07:59:08.708840 25780 tablet_copy_service.cc:140] P 14d8fd1fc3a04d9d967b8033df84f34a: Received BeginTabletCopySession request for tablet 03016ecfa11d4987a2e9d2368a66a5a2 from peer 85fbc6c98f0643469c92215a5cfb2e38 ({username='slave'} at 127.18.13.136:56619)
I20240502 07:59:08.710912 25780 tablet_copy_service.cc:161] P 14d8fd1fc3a04d9d967b8033df84f34a: Beginning new tablet copy session on tablet 03016ecfa11d4987a2e9d2368a66a5a2 from peer 85fbc6c98f0643469c92215a5cfb2e38 at {username='slave'} at 127.18.13.136:56619: session id = 85fbc6c98f0643469c92215a5cfb2e38-03016ecfa11d4987a2e9d2368a66a5a2
I20240502 07:59:08.711997 25780 tablet_copy_source_session.cc:205] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:08.712704 30425 ts_tablet_manager.cc:1398] T 99c3ddedadbf4c0e8858085b66022342 P c6746058199446968d2496df55cdee42: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 07:59:08.713037 30405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03016ecfa11d4987a2e9d2368a66a5a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:08.713181 25780 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session c6746058199446968d2496df55cdee42-99c3ddedadbf4c0e8858085b66022342 received from {username='slave'} at 127.18.13.135:52983
I20240502 07:59:08.713233 25780 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session c6746058199446968d2496df55cdee42-99c3ddedadbf4c0e8858085b66022342 on tablet 99c3ddedadbf4c0e8858085b66022342 with peer c6746058199446968d2496df55cdee42
W20240502 07:59:08.714526 25340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111)
I20240502 07:59:08.715400 30405 tablet_copy_client.cc:783] T 03016ecfa11d4987a2e9d2368a66a5a2 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:08.715553 30405 tablet_copy_client.cc:647] T 03016ecfa11d4987a2e9d2368a66a5a2 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:08.717988 25628 consensus_queue.cc:237] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { promote: true } }
I20240502 07:59:08.719655 27275 raft_consensus.cc:1261] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:08.720085 29256 consensus_queue.cc:1035] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:08.721818 25760 raft_consensus.cc:1261] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Refusing update from remote peer 0dfa00c061c748b18772d5f0566944b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 07:59:08.722204 25564 consensus_peers.cc:485] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 -> Peer e9596bfb8d324b4da93a4288771ed177 (127.18.13.132:45865): Couldn't send request to peer e9596bfb8d324b4da93a4288771ed177. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.722226 29588 consensus_queue.cc:1035] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:08.725000 25758 consensus_queue.cc:237] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
W20240502 07:59:08.726945 25696 consensus_peers.cc:485] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.727061 29588 raft_consensus.cc:2932] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { promote: true } } }
I20240502 07:59:08.728317 25628 raft_consensus.cc:1261] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 2 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240502 07:59:08.728526 25693 consensus_peers.cc:485] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.728844 29321 consensus_queue.cc:1035] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:08.729818 27275 raft_consensus.cc:2932] T fc1ac05112724e0ea1f84f648598304e P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { promote: true } } }
I20240502 07:59:08.731257 29321 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:08.732384 25348 catalog_manager.cc:5484] T fc1ac05112724e0ea1f84f648598304e P 0dfa00c061c748b18772d5f0566944b5 reported cstate change: config changed from index 6 to 7, NON_VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "0dfa00c061c748b18772d5f0566944b5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:08.732482 25628 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 0dfa00c061c748b18772d5f0566944b5 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:08.734532 25341 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5a0605c8c64f4c6fb5e7315a118a0d7f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:08.734877 25347 catalog_manager.cc:5484] T 5a0605c8c64f4c6fb5e7315a118a0d7f P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 8 to 9, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New cstate: current_term: 2 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:08.737233 25477 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a0605c8c64f4c6fb5e7315a118a0d7f with delete_type TABLET_DATA_TOMBSTONED (TS c3be11f0fe304e0f8e008dc1e4c41f70 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:50936
I20240502 07:59:08.738126 25758 raft_consensus.cc:2932] T fc1ac05112724e0ea1f84f648598304e P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e9596bfb8d324b4da93a4288771ed177 (127.18.13.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { promote: true } } }
I20240502 07:59:08.743957 30468 tablet_replica.cc:330] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70: stopping tablet replica
I20240502 07:59:08.744206 30468 raft_consensus.cc:2229] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:08.744426 30468 raft_consensus.cc:2258] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:08.749526 25495 consensus_queue.cc:237] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:08.749931 30468 ts_tablet_manager.cc:1875] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:08.752192 30468 ts_tablet_manager.cc:1888] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240502 07:59:08.752255 30468 log.cc:1196] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c3be11f0fe304e0f8e008dc1e4c41f70: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/5a0605c8c64f4c6fb5e7315a118a0d7f
I20240502 07:59:08.752422 27538 raft_consensus.cc:1261] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 07:59:08.752797 25434 consensus_peers.cc:485] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097): Couldn't send request to peer c5f3003b08fb4214b116c0be8dd62851. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8e50f3d37d842868900a4cff5bf3601. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.752807 25343 catalog_manager.cc:4830] TS c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377): tablet 5a0605c8c64f4c6fb5e7315a118a0d7f (table rebalance_test_table_0 [id=616a1d4f85774bf6b08fc4da5a27702a]) successfully deleted
I20240502 07:59:08.753099 28675 consensus_queue.cc:1035] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 07:59:08.753945 25431 consensus_peers.cc:485] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.769281 28675 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:08.770740 30377 tablet_bootstrap.cc:492] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:08.771032 30377 tablet_bootstrap.cc:492] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: Bootstrap complete.
I20240502 07:59:08.771258 30377 ts_tablet_manager.cc:1367] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: Time spent bootstrapping tablet: real 0.093s user 0.032s sys 0.000s
I20240502 07:59:08.771773 30377 raft_consensus.cc:348] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:08.772049 30377 raft_consensus.cc:729] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5f3003b08fb4214b116c0be8dd62851, State: Initialized, Role: LEARNER
I20240502 07:59:08.772265 30377 consensus_queue.cc:260] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:08.772512 25352 catalog_manager.cc:5484] T d8e50f3d37d842868900a4cff5bf3601 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 9 to 10, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:08.772814 30377 ts_tablet_manager.cc:1398] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:08.773144 27538 raft_consensus.cc:2932] T d8e50f3d37d842868900a4cff5bf3601 P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:08.774384 25911 tablet_copy_service.cc:342] P e9596bfb8d324b4da93a4288771ed177: Request end of tablet copy session c5f3003b08fb4214b116c0be8dd62851-0aa60608089c4f50bcee339d01815a65 received from {username='slave'} at 127.18.13.133:39133
I20240502 07:59:08.774464 25911 tablet_copy_service.cc:433] P e9596bfb8d324b4da93a4288771ed177: ending tablet copy session c5f3003b08fb4214b116c0be8dd62851-0aa60608089c4f50bcee339d01815a65 on tablet 0aa60608089c4f50bcee339d01815a65 with peer c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:08.777192 30154 tablet_bootstrap.cc:492] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c5f3003b08fb4214b116c0be8dd62851: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:08.777551 30154 tablet_bootstrap.cc:492] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c5f3003b08fb4214b116c0be8dd62851: Bootstrap complete.
I20240502 07:59:08.777769 30154 ts_tablet_manager.cc:1367] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c5f3003b08fb4214b116c0be8dd62851: Time spent bootstrapping tablet: real 0.101s user 0.026s sys 0.004s
I20240502 07:59:08.779007 30154 raft_consensus.cc:348] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c5f3003b08fb4214b116c0be8dd62851 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:08.779227 30154 raft_consensus.cc:729] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c5f3003b08fb4214b116c0be8dd62851 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c5f3003b08fb4214b116c0be8dd62851, State: Initialized, Role: LEARNER
I20240502 07:59:08.779388 30154 consensus_queue.cc:260] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c5f3003b08fb4214b116c0be8dd62851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } }
I20240502 07:59:08.780094 30154 ts_tablet_manager.cc:1398] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c5f3003b08fb4214b116c0be8dd62851: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:08.780591 25780 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session c5f3003b08fb4214b116c0be8dd62851-5a0605c8c64f4c6fb5e7315a118a0d7f received from {username='slave'} at 127.18.13.133:39885
I20240502 07:59:08.780674 25780 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session c5f3003b08fb4214b116c0be8dd62851-5a0605c8c64f4c6fb5e7315a118a0d7f on tablet 5a0605c8c64f4c6fb5e7315a118a0d7f with peer c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:08.789418 25758 consensus_queue.cc:237] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:08.791538 25890 raft_consensus.cc:1261] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 07:59:08.791684 25694 consensus_peers.cc:485] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f21d6014b91c45df96325017b57f19e3. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.792369 25495 raft_consensus.cc:1261] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:08.792672 29321 consensus_queue.cc:1035] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:08.793375 27275 raft_consensus.cc:1261] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 07:59:08.793928 25693 consensus_peers.cc:485] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.795888 28880 consensus_queue.cc:1035] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:08.796125 28880 consensus_queue.cc:1035] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:08.798185 28960 raft_consensus.cc:2932] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:08.798660 25497 raft_consensus.cc:2932] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:08.801993 25890 raft_consensus.cc:2932] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:08.802641 25352 catalog_manager.cc:5484] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 5 to 6, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:08.805562 30450 tablet_replica.cc:330] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5: stopping tablet replica
I20240502 07:59:08.805701 30450 raft_consensus.cc:2229] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:08.805893 30450 raft_consensus.cc:2258] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 07:59:08.806012 25340 catalog_manager.cc:4573] TS a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 843d1c1d6384476983749a78f65a5c74: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111)
I20240502 07:59:08.806299 30400 tablet_copy_client.cc:523] T eb0c33c3e4014eb3b931d64adac60e1a P c6746058199446968d2496df55cdee42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:08.806715 30450 ts_tablet_manager.cc:1875] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:08.808104 30400 tablet_bootstrap.cc:492] T eb0c33c3e4014eb3b931d64adac60e1a P c6746058199446968d2496df55cdee42: Bootstrap starting.
I20240502 07:59:08.810243 30450 ts_tablet_manager.cc:1888] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 07:59:08.810348 30450 log.cc:1196] T 843d1c1d6384476983749a78f65a5c74 P 0dfa00c061c748b18772d5f0566944b5: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/843d1c1d6384476983749a78f65a5c74
I20240502 07:59:08.811057 25343 catalog_manager.cc:4830] TS 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130:38825): tablet 843d1c1d6384476983749a78f65a5c74 (table rebalance_test_table_9 [id=10514d1b971c4d5c9e2b1f433f40d2c1]) successfully deleted
W20240502 07:59:08.811645 27604 consensus_peers.cc:485] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.820045 30405 tablet_copy_client.cc:523] T 03016ecfa11d4987a2e9d2368a66a5a2 P 85fbc6c98f0643469c92215a5cfb2e38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:08.821506 30405 tablet_bootstrap.cc:492] T 03016ecfa11d4987a2e9d2368a66a5a2 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap starting.
I20240502 07:59:08.841974 27723 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskeLuhys/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 07:59:02.265880 27728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:02.267050 27728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:02.274883 27728 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:59:03.524179 27728 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 07:59:04.245828 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.245920 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:04.245952 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.245983 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.246017 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:04.246080 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.246119 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8aaedf9499044a93ab9419f10e2342e2: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.246147 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f72d1ad6e00b46cf99f485a9ef60b9b1: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.246176 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.246203 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.246237 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7c7fb9171c9548a7ad2d45e295d780cb: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:04.246266 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.246304 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f72d1ad6e00b46cf99f485a9ef60b9b1: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.246332 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:04.246361 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.246395 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.246428 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 616a1d4f85774bf6b08fc4da5a27702a: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.246455 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.246485 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:04.315382 27728 rebalancer_tool.cc:1074] tablet 87d6266008784883891ca32c01363dbe: '0dfa00c061c748b18772d5f0566944b5' -> 'c5f3003b08fb4214b116c0be8dd62851' move scheduled
I20240502 07:59:04.421263 27728 rebalancer_tool.cc:1074] tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0: 'e9596bfb8d324b4da93a4288771ed177' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:04.500557 27728 rebalancer_tool.cc:1074] tablet 8904fa47fc274c14acd291b4c91964be: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> '85fbc6c98f0643469c92215a5cfb2e38' move scheduled
I20240502 07:59:04.587182 27728 rebalancer_tool.cc:1074] tablet fd68090fdadf45faa0e3b2936b30047e: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:04.733875 27728 rebalancer_tool.cc:1074] tablet 03016ecfa11d4987a2e9d2368a66a5a2: '0dfa00c061c748b18772d5f0566944b5' -> 'c5f3003b08fb4214b116c0be8dd62851' move scheduled
I20240502 07:59:04.838615 27728 rebalancer_tool.cc:1084] tablet c81a7795712e45b5a7fd97450a9f06a7: 'e9596bfb8d324b4da93a4288771ed177' -> '85fbc6c98f0643469c92215a5cfb2e38' move ignored: Illegal state: tablet c81a7795712e45b5a7fd97450a9f06a7: replica 85fbc6c98f0643469c92215a5cfb2e38 is already present
I20240502 07:59:04.838716 27728 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:06.282763 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c6746058199446968d2496df55cdee42
W20240502 07:59:06.282960 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7c7fb9171c9548a7ad2d45e295d780cb: could not find any suitable replica to move from server c6746058199446968d2496df55cdee42 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:06.283016 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c6746058199446968d2496df55cdee42
W20240502 07:59:06.283069 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8aaedf9499044a93ab9419f10e2342e2: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:06.283111 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:06.283152 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 616a1d4f85774bf6b08fc4da5a27702a: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c6746058199446968d2496df55cdee42
W20240502 07:59:06.283205 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:06.283247 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c6746058199446968d2496df55cdee42
W20240502 07:59:06.283288 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f72d1ad6e00b46cf99f485a9ef60b9b1: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:06.283325 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7c7fb9171c9548a7ad2d45e295d780cb: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:06.283366 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:06.283403 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:06.424226 27728 rebalancer_tool.cc:1074] tablet ff31ddd6b3d74a9aae6617afb56c40f5: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> '' move scheduled
I20240502 07:59:06.489223 27728 rebalancer_tool.cc:1084] tablet a1fa713b432740d98bcab918d251b54d: 'e9596bfb8d324b4da93a4288771ed177' -> 'a17bbff6a71b4667b14f5dd12bca3057' move ignored: Not found: Peer with uuid e9596bfb8d324b4da93a4288771ed177 not found in consensus config
I20240502 07:59:06.489334 27728 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:06.994902 29524 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.18.13.190:46035, user_credentials={real_user=slave}} blocked reactor thread for 67303.6us
W20240502 07:59:07.262634 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7c7fb9171c9548a7ad2d45e295d780cb: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:07.262835 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:07.262949 27728 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c6746058199446968d2496df55cdee42
I20240502 07:59:07.298612 27728 rebalancer_tool.cc:1074] tablet f21d6014b91c45df96325017b57f19e3: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:07.837697 27728 rebalancer_tool.cc:1074] tablet 5a0605c8c64f4c6fb5e7315a118a0d7f: 'a17bbff6a71b4667b14f5dd12bca3057' -> '' move scheduled
I20240502 07:59:07.926101 27728 rebalancer_tool.cc:1074] tablet eb0c33c3e4014eb3b931d64adac60e1a: '0dfa00c061c748b18772d5f0566944b5' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:07.971911 27728 rebalancer_tool.cc:1084] tablet dbfd23aa383f413cbe4a5ccd8146d2b6: 'e9596bfb8d324b4da93a4288771ed177' -> '85fbc6c98f0643469c92215a5cfb2e38' move ignored: Illegal state: tablet dbfd23aa383f413cbe4a5ccd8146d2b6: replica 85fbc6c98f0643469c92215a5cfb2e38 is already present
I20240502 07:59:07.972002 27728 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:08.552626 30319 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 07:59:08.314252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.13.134:43037 (local address 127.0.0.1:43626)
0502 07:59:08.314619 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0502 07:59:08.314630 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0502 07:59:08.314684 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 07:59:08.551347 (+236663us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":310}
Illegal state: tablet server a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): unacceptable health status UNAVAILABLE
I20240502 07:59:08.846905 28515 raft_consensus.cc:1050] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70: attempting to promote NON_VOTER 0dfa00c061c748b18772d5f0566944b5 to VOTER
I20240502 07:59:08.847340 28515 consensus_queue.cc:237] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: false } }
I20240502 07:59:08.849488 27538 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 07:59:08.849701 29321 raft_consensus.cc:1050] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a: attempting to promote NON_VOTER c5f3003b08fb4214b116c0be8dd62851 to VOTER
I20240502 07:59:08.850085 29109 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:59:08.850220 29321 consensus_queue.cc:237] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:08.850876 27670 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 07:59:08.851701 25628 raft_consensus.cc:1261] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEARNER]: Refusing update from remote peer c3be11f0fe304e0f8e008dc1e4c41f70: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 07:59:08.852253 28375 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 07:59:08.852890 25694 consensus_peers.cc:485] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136:38043): Couldn't send request to peer 85fbc6c98f0643469c92215a5cfb2e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f21d6014b91c45df96325017b57f19e3. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.853821 29109 consensus_queue.cc:1035] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:59:08.856091 25890 raft_consensus.cc:1261] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:08.856719 27274 raft_consensus.cc:1261] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 07:59:08.857206 25693 consensus_peers.cc:485] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Couldn't send request to peer a17bbff6a71b4667b14f5dd12bca3057. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:08.857105 27275 raft_consensus.cc:2932] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:08.857311 25497 raft_consensus.cc:1261] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:08.857700 30494 consensus_queue.cc:1035] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:08.858603 28375 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: false } } }
I20240502 07:59:08.859558 28960 consensus_queue.cc:1035] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:08.863322 28960 consensus_queue.cc:1035] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:08.863627 25353 catalog_manager.cc:5484] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 11 to 12, 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:08.863616 28960 raft_consensus.cc:2932] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:08.866320 25890 raft_consensus.cc:2932] T f21d6014b91c45df96325017b57f19e3 P e9596bfb8d324b4da93a4288771ed177 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:08.866423 27538 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: false } } }
I20240502 07:59:08.868438 25628 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: false } } }
I20240502 07:59:08.868847 25348 catalog_manager.cc:5484] T f21d6014b91c45df96325017b57f19e3 P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:08.871265 25497 raft_consensus.cc:2932] T f21d6014b91c45df96325017b57f19e3 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:08.872359 27670 raft_consensus.cc:2932] T ee25634bf534443b90fdbdcebdbcb02e P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: false } } }
I20240502 07:59:08.874902 27275 raft_consensus.cc:2932] T f21d6014b91c45df96325017b57f19e3 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } } }
I20240502 07:59:08.899458 30400 tablet_bootstrap.cc:492] T eb0c33c3e4014eb3b931d64adac60e1a P c6746058199446968d2496df55cdee42: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:08.900189 30400 tablet_bootstrap.cc:492] T eb0c33c3e4014eb3b931d64adac60e1a P c6746058199446968d2496df55cdee42: Bootstrap complete.
I20240502 07:59:08.900521 30400 ts_tablet_manager.cc:1367] T eb0c33c3e4014eb3b931d64adac60e1a P c6746058199446968d2496df55cdee42: Time spent bootstrapping tablet: real 0.092s user 0.034s sys 0.000s
I20240502 07:59:08.901080 30400 raft_consensus.cc:348] T eb0c33c3e4014eb3b931d64adac60e1a P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:08.901326 30400 raft_consensus.cc:729] T eb0c33c3e4014eb3b931d64adac60e1a P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6746058199446968d2496df55cdee42, State: Initialized, Role: LEARNER
I20240502 07:59:08.901484 30400 consensus_queue.cc:260] T eb0c33c3e4014eb3b931d64adac60e1a P c6746058199446968d2496df55cdee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:08.901965 30400 ts_tablet_manager.cc:1398] T eb0c33c3e4014eb3b931d64adac60e1a P c6746058199446968d2496df55cdee42: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:08.902185 27724 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskeLuhys/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 07:59:02.342389 27726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:02.342713 27726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:02.347657 27726 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:59:03.287125 27726 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 07:59:03.688143 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c6746058199446968d2496df55cdee42
W20240502 07:59:03.688391 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:03.688444 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:03.688539 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8aaedf9499044a93ab9419f10e2342e2: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:03.688601 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:03.688652 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:03.688724 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:03.688774 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 616a1d4f85774bf6b08fc4da5a27702a: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:03.688819 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f72d1ad6e00b46cf99f485a9ef60b9b1: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c6746058199446968d2496df55cdee42
W20240502 07:59:03.688864 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7c7fb9171c9548a7ad2d45e295d780cb: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c6746058199446968d2496df55cdee42
W20240502 07:59:03.688930 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:03.688978 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c6746058199446968d2496df55cdee42
W20240502 07:59:03.689023 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:03.689069 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:03.689116 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server 85fbc6c98f0643469c92215a5cfb2e38
I20240502 07:59:03.742650 27726 rebalancer_tool.cc:1074] tablet a1fa713b432740d98bcab918d251b54d: 'e9596bfb8d324b4da93a4288771ed177' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:03.798718 27726 rebalancer_tool.cc:1074] tablet ff31ddd6b3d74a9aae6617afb56c40f5: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:03.983922 27726 rebalancer_tool.cc:1074] tablet 71a67cb0f6b54bfb9eb44400fe21ce40: '0dfa00c061c748b18772d5f0566944b5' -> '85fbc6c98f0643469c92215a5cfb2e38' move scheduled
I20240502 07:59:04.072535 27726 rebalancer_tool.cc:1074] tablet eb0c33c3e4014eb3b931d64adac60e1a: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'c5f3003b08fb4214b116c0be8dd62851' move scheduled
I20240502 07:59:04.119334 27726 rebalancer_tool.cc:1074] tablet 21a8d5798c8943658113d67c88f4ad25: 'e9596bfb8d324b4da93a4288771ed177' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:04.159431 27726 rebalancer_tool.cc:1074] tablet 0aa60608089c4f50bcee339d01815a65: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:04.186887 27726 rebalancer_tool.cc:1084] tablet 056ba8d15d18409e946c905b8876097a: '0dfa00c061c748b18772d5f0566944b5' -> '85fbc6c98f0643469c92215a5cfb2e38' move ignored: Illegal state: tablet 056ba8d15d18409e946c905b8876097a: replica 85fbc6c98f0643469c92215a5cfb2e38 is already present
I20240502 07:59:04.187003 27726 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:04.844705 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.844818 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.844868 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7c7fb9171c9548a7ad2d45e295d780cb: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.844915 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.844995 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:04.845052 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.845094 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f72d1ad6e00b46cf99f485a9ef60b9b1: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.845158 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8aaedf9499044a93ab9419f10e2342e2: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.845217 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f72d1ad6e00b46cf99f485a9ef60b9b1: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.845257 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8aaedf9499044a93ab9419f10e2342e2: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:04.845304 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.845346 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:04.845391 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 616a1d4f85774bf6b08fc4da5a27702a: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.845432 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.845495 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:04.845539 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.845577 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.845616 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:05.004218 27726 rebalancer_tool.cc:1074] tablet a3bc416015dc40d88743215d2cf46230: '0dfa00c061c748b18772d5f0566944b5' -> '85fbc6c98f0643469c92215a5cfb2e38' move scheduled
I20240502 07:59:05.188645 27726 rebalancer_tool.cc:1074] tablet 5eb2de2942914800bdbe924d45a9b1e7: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:05.421346 27726 rebalancer_tool.cc:1074] tablet fc1ac05112724e0ea1f84f648598304e: 'e9596bfb8d324b4da93a4288771ed177' -> 'c5f3003b08fb4214b116c0be8dd62851' move scheduled
I20240502 07:59:05.671310 27726 rebalancer_tool.cc:1074] tablet 99c3ddedadbf4c0e8858085b66022342: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> '85fbc6c98f0643469c92215a5cfb2e38' move scheduled
I20240502 07:59:05.896273 27726 rebalancer_tool.cc:1074] tablet d8e50f3d37d842868900a4cff5bf3601: 'e9596bfb8d324b4da93a4288771ed177' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:05.989287 27726 rebalancer_tool.cc:1074] tablet 843d1c1d6384476983749a78f65a5c74: '0dfa00c061c748b18772d5f0566944b5' -> 'c5f3003b08fb4214b116c0be8dd62851' move scheduled
I20240502 07:59:06.049240 27726 rebalancer_tool.cc:1084] tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0: '14d8fd1fc3a04d9d967b8033df84f34a' -> '85fbc6c98f0643469c92215a5cfb2e38' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } }
I20240502 07:59:06.049320 27726 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:07.142524 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c6746058199446968d2496df55cdee42
W20240502 07:59:07.226720 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server c6746058199446968d2496df55cdee42 to server 14d8fd1fc3a04d9d967b8033df84f34a
W20240502 07:59:07.226848 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f72d1ad6e00b46cf99f485a9ef60b9b1: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c6746058199446968d2496df55cdee42
W20240502 07:59:07.226939 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:07.227023 27726 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:07.241590 27726 rebalancer_tool.cc:1074] tablet 87d6266008784883891ca32c01363dbe: '14d8fd1fc3a04d9d967b8033df84f34a' -> '' move scheduled
I20240502 07:59:07.254686 27726 rebalancer_tool.cc:1074] tablet 8904fa47fc274c14acd291b4c91964be: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> '' move scheduled
I20240502 07:59:07.321013 27726 rebalancer_tool.cc:1074] tablet ee25634bf534443b90fdbdcebdbcb02e: 'e9596bfb8d324b4da93a4288771ed177' -> '' move scheduled
I20240502 07:59:07.348965 27726 rebalancer_tool.cc:1074] tablet 1ad22d2e464d4defab6bd0061e6058f8: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:07.384313 27726 rebalancer_tool.cc:1074] tablet f21d6014b91c45df96325017b57f19e3: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'c5f3003b08fb4214b116c0be8dd62851' move scheduled
I20240502 07:59:07.432843 27726 rebalancer_tool.cc:1074] tablet dbfd23aa383f413cbe4a5ccd8146d2b6: '0dfa00c061c748b18772d5f0566944b5' -> '85fbc6c98f0643469c92215a5cfb2e38' move scheduled
I20240502 07:59:07.456396 27726 rebalancer_tool.cc:1074] tablet e5958bb5c8fc4c1d91ae8f1e8b3861d0: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:07.565688 27726 rebalancer_tool.cc:1136] tablet 0aa60608089c4f50bcee339d01815a65: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'a17bbff6a71b4667b14f5dd12bca3057' move completed: OK
I20240502 07:59:07.582957 27726 rebalancer_tool.cc:1136] tablet 21a8d5798c8943658113d67c88f4ad25: 'e9596bfb8d324b4da93a4288771ed177' -> 'c6746058199446968d2496df55cdee42' move completed: OK
I20240502 07:59:07.594724 27726 rebalancer_tool.cc:1136] tablet eb0c33c3e4014eb3b931d64adac60e1a: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'c5f3003b08fb4214b116c0be8dd62851' move completed: OK
I20240502 07:59:07.606678 27726 rebalancer_tool.cc:1136] tablet 71a67cb0f6b54bfb9eb44400fe21ce40: '0dfa00c061c748b18772d5f0566944b5' -> '85fbc6c98f0643469c92215a5cfb2e38' move completed: OK
I20240502 07:59:07.713194 27726 rebalancer_tool.cc:1136] tablet fc1ac05112724e0ea1f84f648598304e: 'e9596bfb8d324b4da93a4288771ed177' -> 'c5f3003b08fb4214b116c0be8dd62851' move completed: OK
I20240502 07:59:07.730578 27726 rebalancer_tool.cc:1136] tablet a3bc416015dc40d88743215d2cf46230: '0dfa00c061c748b18772d5f0566944b5' -> '85fbc6c98f0643469c92215a5cfb2e38' move completed: OK
I20240502 07:59:07.769145 27726 rebalancer_tool.cc:1136] tablet 5eb2de2942914800bdbe924d45a9b1e7: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'a17bbff6a71b4667b14f5dd12bca3057' move completed: Incomplete: tablet 5eb2de2942914800bdbe924d45a9b1e7, TS 14d8fd1fc3a04d9d967b8033df84f34a -> TS a17bbff6a71b4667b14f5dd12bca3057 move failed, target replica disappeared
I20240502 07:59:08.068307 27726 rebalancer_tool.cc:1136] tablet 99c3ddedadbf4c0e8858085b66022342: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> '85fbc6c98f0643469c92215a5cfb2e38' move completed: OK
I20240502 07:59:08.190379 27726 rebalancer_tool.cc:1136] tablet a1fa713b432740d98bcab918d251b54d: 'e9596bfb8d324b4da93a4288771ed177' -> 'c6746058199446968d2496df55cdee42' move completed: OK
I20240502 07:59:08.249778 27726 rebalancer_tool.cc:1136] tablet d8e50f3d37d842868900a4cff5bf3601: 'e9596bfb8d324b4da93a4288771ed177' -> 'c6746058199446968d2496df55cdee42' move completed: OK
I20240502 07:59:08.263770 27726 rebalancer_tool.cc:1136] tablet 87d6266008784883891ca32c01363dbe: '14d8fd1fc3a04d9d967b8033df84f34a' -> '' move completed: OK
I20240502 07:59:08.273341 27726 rebalancer_tool.cc:1084] tablet 03016ecfa11d4987a2e9d2368a66a5a2: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> '85fbc6c98f0643469c92215a5cfb2e38' move ignored: Illegal state: tablet 03016ecfa11d4987a2e9d2368a66a5a2: replica 85fbc6c98f0643469c92215a5cfb2e38 is already present
I20240502 07:59:08.273440 27726 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:08.552670 30387 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 07:59:08.442019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.13.134:43037 (local address 127.0.0.1:43628)
0502 07:59:08.442235 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0502 07:59:08.442252 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0502 07:59:08.442339 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 07:59:08.552102 (+109763us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
Illegal state: tablet server a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): unacceptable health status UNAVAILABLE
I20240502 07:59:08.902922 25517 tablet_copy_service.cc:342] P c3be11f0fe304e0f8e008dc1e4c41f70: Request end of tablet copy session c6746058199446968d2496df55cdee42-eb0c33c3e4014eb3b931d64adac60e1a received from {username='slave'} at 127.18.13.135:42665
I20240502 07:59:08.903005 25517 tablet_copy_service.cc:433] P c3be11f0fe304e0f8e008dc1e4c41f70: ending tablet copy session c6746058199446968d2496df55cdee42-eb0c33c3e4014eb3b931d64adac60e1a on tablet eb0c33c3e4014eb3b931d64adac60e1a with peer c6746058199446968d2496df55cdee42
I20240502 07:59:08.909454 27722 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskeLuhys/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 07:59:02.260712 27725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:02.261090 27725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:02.266491 27725 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:59:03.211660 27725 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 07:59:03.736399 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:03.736620 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f72d1ad6e00b46cf99f485a9ef60b9b1: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:03.736721 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:03.736771 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c6746058199446968d2496df55cdee42
W20240502 07:59:03.736879 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:03.736945 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:03.736994 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c6746058199446968d2496df55cdee42
W20240502 07:59:03.737041 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 616a1d4f85774bf6b08fc4da5a27702a: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:03.737087 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7c7fb9171c9548a7ad2d45e295d780cb: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:03.737152 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8aaedf9499044a93ab9419f10e2342e2: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:03.737200 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:03.737260 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:03.737306 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c6746058199446968d2496df55cdee42
W20240502 07:59:03.737349 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:03.768797 27725 rebalancer_tool.cc:1074] tablet d6929d2560404acb9bb53f54a181f958: '0dfa00c061c748b18772d5f0566944b5' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:03.789659 27725 rebalancer_tool.cc:1084] tablet ff31ddd6b3d74a9aae6617afb56c40f5: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'c5f3003b08fb4214b116c0be8dd62851' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } }
I20240502 07:59:03.789778 27725 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:04.345288 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 616a1d4f85774bf6b08fc4da5a27702a: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.345396 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8aaedf9499044a93ab9419f10e2342e2: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.345496 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7c7fb9171c9548a7ad2d45e295d780cb: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.345542 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.345582 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.345628 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.345666 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 616a1d4f85774bf6b08fc4da5a27702a: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:04.345713 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.345744 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.345784 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f72d1ad6e00b46cf99f485a9ef60b9b1: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.345844 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.345916 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f72d1ad6e00b46cf99f485a9ef60b9b1: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.345968 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:04.346004 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.346040 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.346071 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.346256 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 616a1d4f85774bf6b08fc4da5a27702a: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.346323 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:04.346355 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8aaedf9499044a93ab9419f10e2342e2: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c6746058199446968d2496df55cdee42
I20240502 07:59:04.387049 27725 rebalancer_tool.cc:1074] tablet a1fa713b432740d98bcab918d251b54d: 'e9596bfb8d324b4da93a4288771ed177' -> 'c5f3003b08fb4214b116c0be8dd62851' move scheduled
I20240502 07:59:04.423569 27725 rebalancer_tool.cc:1074] tablet ff31ddd6b3d74a9aae6617afb56c40f5: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:04.476923 27725 rebalancer_tool.cc:1084] tablet 87d6266008784883891ca32c01363dbe: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'c5f3003b08fb4214b116c0be8dd62851' move ignored: Illegal state: tablet 87d6266008784883891ca32c01363dbe: replica c5f3003b08fb4214b116c0be8dd62851 is already present
I20240502 07:59:04.477012 27725 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:05.230075 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c6746058199446968d2496df55cdee42
W20240502 07:59:05.231195 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:05.233892 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c6746058199446968d2496df55cdee42
W20240502 07:59:05.233978 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c6746058199446968d2496df55cdee42
W20240502 07:59:05.234016 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8aaedf9499044a93ab9419f10e2342e2: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:05.234051 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:05.234083 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:05.234113 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:05.234175 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7c7fb9171c9548a7ad2d45e295d780cb: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:05.234206 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:05.234246 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:05.234282 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 616a1d4f85774bf6b08fc4da5a27702a: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:05.234313 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c6746058199446968d2496df55cdee42
W20240502 07:59:05.234345 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:05.234388 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c6746058199446968d2496df55cdee42
I20240502 07:59:05.378358 27725 rebalancer_tool.cc:1074] tablet fd68090fdadf45faa0e3b2936b30047e: '14d8fd1fc3a04d9d967b8033df84f34a' -> '85fbc6c98f0643469c92215a5cfb2e38' move scheduled
I20240502 07:59:06.081405 27725 rebalancer_tool.cc:1074] tablet dbfd23aa383f413cbe4a5ccd8146d2b6: 'e9596bfb8d324b4da93a4288771ed177' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:06.240350 27725 rebalancer_tool.cc:1074] tablet 0aa60608089c4f50bcee339d01815a65: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:06.345978 27725 rebalancer_tool.cc:1084] tablet 843d1c1d6384476983749a78f65a5c74: '0dfa00c061c748b18772d5f0566944b5' -> 'c5f3003b08fb4214b116c0be8dd62851' move ignored: Illegal state: tablet 843d1c1d6384476983749a78f65a5c74: replica c5f3003b08fb4214b116c0be8dd62851 is already present
I20240502 07:59:06.346042 27725 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:07.264729 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server 14d8fd1fc3a04d9d967b8033df84f34a
W20240502 07:59:07.264880 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:07.264932 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:07.308812 27725 rebalancer_tool.cc:1074] tablet 5a0605c8c64f4c6fb5e7315a118a0d7f: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> '' move scheduled
I20240502 07:59:07.413210 27725 rebalancer_tool.cc:1074] tablet 9722dae4e4414084bdd8066b41bcc47a: '14d8fd1fc3a04d9d967b8033df84f34a' -> '0dfa00c061c748b18772d5f0566944b5' move scheduled
I20240502 07:59:07.481102 27725 rebalancer_tool.cc:1074] tablet 037fed57490949bbbafa11ebb44d4bb8: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'c5f3003b08fb4214b116c0be8dd62851' move scheduled
I20240502 07:59:07.514326 27725 rebalancer_tool.cc:1074] tablet 03016ecfa11d4987a2e9d2368a66a5a2: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:07.542716 27725 rebalancer_tool.cc:1074] tablet eb0c33c3e4014eb3b931d64adac60e1a: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:07.601071 27725 rebalancer_tool.cc:1084] tablet f21d6014b91c45df96325017b57f19e3: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'a17bbff6a71b4667b14f5dd12bca3057' move ignored: Illegal state: tablet f21d6014b91c45df96325017b57f19e3: replica a17bbff6a71b4667b14f5dd12bca3057 is already present
I20240502 07:59:07.601174 27725 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:08.067153 27725 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 616a1d4f85774bf6b08fc4da5a27702a: could not find any suitable replica to move from server a17bbff6a71b4667b14f5dd12bca3057 to server c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:08.230942 27725 rebalancer_tool.cc:1084] tablet 99c3ddedadbf4c0e8858085b66022342: '0dfa00c061c748b18772d5f0566944b5' -> 'a17bbff6a71b4667b14f5dd12bca3057' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { replace: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:08.231035 27725 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:08.552639 30401 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 07:59:08.472392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.13.134:43037 (local address 127.0.0.1:43638)
0502 07:59:08.482361 (+ 9969us) negotiation.cc:107] Waiting for socket to connect
0502 07:59:08.482375 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0502 07:59:08.482440 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 07:59:08.552070 (+ 69630us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":9910,"thread_start_us":83,"threads_started":1}
Illegal state: tablet server a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): unacceptable health status UNAVAILABLE
I20240502 07:59:08.922190 30405 tablet_bootstrap.cc:492] T 03016ecfa11d4987a2e9d2368a66a5a2 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:08.922528 30405 tablet_bootstrap.cc:492] T 03016ecfa11d4987a2e9d2368a66a5a2 P 85fbc6c98f0643469c92215a5cfb2e38: Bootstrap complete.
I20240502 07:59:08.922742 30405 ts_tablet_manager.cc:1367] T 03016ecfa11d4987a2e9d2368a66a5a2 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent bootstrapping tablet: real 0.101s user 0.034s sys 0.000s
I20240502 07:59:08.923131 30405 raft_consensus.cc:348] T 03016ecfa11d4987a2e9d2368a66a5a2 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:08.923287 30405 raft_consensus.cc:729] T 03016ecfa11d4987a2e9d2368a66a5a2 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 85fbc6c98f0643469c92215a5cfb2e38, State: Initialized, Role: LEARNER
I20240502 07:59:08.923393 30405 consensus_queue.cc:260] T 03016ecfa11d4987a2e9d2368a66a5a2 P 85fbc6c98f0643469c92215a5cfb2e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: true } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: NON_VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: true } }
I20240502 07:59:08.923684 30405 ts_tablet_manager.cc:1398] T 03016ecfa11d4987a2e9d2368a66a5a2 P 85fbc6c98f0643469c92215a5cfb2e38: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:08.924296 25780 tablet_copy_service.cc:342] P 14d8fd1fc3a04d9d967b8033df84f34a: Request end of tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-03016ecfa11d4987a2e9d2368a66a5a2 received from {username='slave'} at 127.18.13.136:56619
I20240502 07:59:08.924389 25780 tablet_copy_service.cc:433] P 14d8fd1fc3a04d9d967b8033df84f34a: ending tablet copy session 85fbc6c98f0643469c92215a5cfb2e38-03016ecfa11d4987a2e9d2368a66a5a2 on tablet 03016ecfa11d4987a2e9d2368a66a5a2 with peer 85fbc6c98f0643469c92215a5cfb2e38
I20240502 07:59:08.952507 30262 raft_consensus.cc:1050] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177: attempting to promote NON_VOTER c5f3003b08fb4214b116c0be8dd62851 to VOTER
I20240502 07:59:08.952972 30262 consensus_queue.cc:237] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:08.954504 25628 raft_consensus.cc:1261] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:08.954603 27275 raft_consensus.cc:1261] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:08.954970 25497 raft_consensus.cc:1261] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:08.955399 26712 consensus_queue.cc:1035] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:08.955621 29169 consensus_queue.cc:1035] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:08.955824 29168 consensus_queue.cc:1035] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:08.958374 29169 raft_consensus.cc:2932] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:08.959055 25497 raft_consensus.cc:2932] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:08.960698 25628 raft_consensus.cc:2932] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:08.961608 25347 catalog_manager.cc:5484] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70 reported cstate change: config changed from index 4 to 5, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:08.963409 27670 raft_consensus.cc:1203] T dbfd23aa383f413cbe4a5ccd8146d2b6 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 07:59:08.965195 27275 raft_consensus.cc:2932] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:08.966383 25890 consensus_queue.cc:237] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } }
I20240502 07:59:08.969362 29177 raft_consensus.cc:2932] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:08.969913 27275 raft_consensus.cc:1261] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Refusing update from remote peer e9596bfb8d324b4da93a4288771ed177: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:08.970378 30544 consensus_queue.cc:1035] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:08.970795 25342 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 037fed57490949bbbafa11ebb44d4bb8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:08.972153 25628 raft_consensus.cc:2932] T 037fed57490949bbbafa11ebb44d4bb8 P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:08.972179 25348 catalog_manager.cc:5484] T 037fed57490949bbbafa11ebb44d4bb8 P e9596bfb8d324b4da93a4288771ed177 reported cstate change: config changed from index 5 to 6, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "e9596bfb8d324b4da93a4288771ed177" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:08.972626 27538 raft_consensus.cc:1203] T eb0c33c3e4014eb3b931d64adac60e1a P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240502 07:59:08.972695 27275 raft_consensus.cc:2932] T 037fed57490949bbbafa11ebb44d4bb8 P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } }
I20240502 07:59:08.976104 25477 tablet_service.cc:1514] Processing DeleteTablet for tablet 037fed57490949bbbafa11ebb44d4bb8 with delete_type TABLET_DATA_TOMBSTONED (TS c3be11f0fe304e0f8e008dc1e4c41f70 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50936
I20240502 07:59:08.976361 30468 tablet_replica.cc:330] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70: stopping tablet replica
I20240502 07:59:08.976487 30468 raft_consensus.cc:2229] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:08.976613 30468 raft_consensus.cc:2258] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:08.977568 30468 ts_tablet_manager.cc:1875] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:08.979351 30468 ts_tablet_manager.cc:1888] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:08.979429 30468 log.cc:1196] T 037fed57490949bbbafa11ebb44d4bb8 P c3be11f0fe304e0f8e008dc1e4c41f70: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/037fed57490949bbbafa11ebb44d4bb8
I20240502 07:59:08.979892 25343 catalog_manager.cc:4830] TS c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377): tablet 037fed57490949bbbafa11ebb44d4bb8 (table rebalance_test_table_8 [id=8aaedf9499044a93ab9419f10e2342e2]) successfully deleted
I20240502 07:59:08.989861 25497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ee25634bf534443b90fdbdcebdbcb02e"
dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44246
I20240502 07:59:08.989995 25497 raft_consensus.cc:595] T ee25634bf534443b90fdbdcebdbcb02e P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:09.007968 27275 raft_consensus.cc:1203] T 0aa60608089c4f50bcee339d01815a65 P c5f3003b08fb4214b116c0be8dd62851 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240502 07:59:09.008100 25628 raft_consensus.cc:1203] T 71a67cb0f6b54bfb9eb44400fe21ce40 P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 07:59:09.031603 25497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb0c33c3e4014eb3b931d64adac60e1a"
dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44264
I20240502 07:59:09.031750 25497 raft_consensus.cc:595] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:09.050818 28375 raft_consensus.cc:979] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70: : Instructing follower 85fbc6c98f0643469c92215a5cfb2e38 to start an election
I20240502 07:59:09.050949 28375 raft_consensus.cc:1067] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Signalling peer 85fbc6c98f0643469c92215a5cfb2e38 to start an election
I20240502 07:59:09.051438 27670 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb0c33c3e4014eb3b931d64adac60e1a"
dest_uuid: "85fbc6c98f0643469c92215a5cfb2e38"
from {username='slave'} at 127.18.13.129:57955
I20240502 07:59:09.051590 27670 raft_consensus.cc:482] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:09.051654 27670 raft_consensus.cc:3037] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:09.052735 27670 raft_consensus.cc:504] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:09.053194 27670 leader_election.cc:288] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [CANDIDATE]: Term 2 election: Requested vote from peers c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377), c5f3003b08fb4214b116c0be8dd62851 (127.18.13.133:39097)
I20240502 07:59:09.053967 25497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0c33c3e4014eb3b931d64adac60e1a" candidate_uuid: "85fbc6c98f0643469c92215a5cfb2e38" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70"
I20240502 07:59:09.054113 25497 raft_consensus.cc:3032] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:09.054157 25497 raft_consensus.cc:729] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c3be11f0fe304e0f8e008dc1e4c41f70, State: Running, Role: LEADER
I20240502 07:59:09.054278 25497 consensus_queue.cc:260] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:09.054430 25497 raft_consensus.cc:3037] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:09.055373 25497 raft_consensus.cc:2454] T eb0c33c3e4014eb3b931d64adac60e1a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85fbc6c98f0643469c92215a5cfb2e38 in term 2.
I20240502 07:59:09.055685 27607 leader_election.cc:302] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85fbc6c98f0643469c92215a5cfb2e38, c3be11f0fe304e0f8e008dc1e4c41f70; no voters:
I20240502 07:59:09.055907 30429 raft_consensus.cc:2781] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:09.055996 30429 raft_consensus.cc:686] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [term 2 LEADER]: Becoming Leader. State: Replica: 85fbc6c98f0643469c92215a5cfb2e38, State: Running, Role: LEADER
I20240502 07:59:09.056131 30429 consensus_queue.cc:237] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } }
I20240502 07:59:09.056270 27538 raft_consensus.cc:1203] T 99c3ddedadbf4c0e8858085b66022342 P c6746058199446968d2496df55cdee42 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 07:59:09.058176 25347 catalog_manager.cc:5484] T eb0c33c3e4014eb3b931d64adac60e1a P 85fbc6c98f0643469c92215a5cfb2e38 reported cstate change: term changed from 1 to 2, leader changed from c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) to 85fbc6c98f0643469c92215a5cfb2e38 (127.18.13.136). New cstate: current_term: 2 leader_uuid: "85fbc6c98f0643469c92215a5cfb2e38" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85fbc6c98f0643469c92215a5cfb2e38" member_type: VOTER last_known_addr { host: "127.18.13.136" port: 38043 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: NON_VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:09.060590 27275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0c33c3e4014eb3b931d64adac60e1a" candidate_uuid: "85fbc6c98f0643469c92215a5cfb2e38" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "c5f3003b08fb4214b116c0be8dd62851"
I20240502 07:59:09.060717 27275 raft_consensus.cc:3037] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:09.061636 27275 raft_consensus.cc:2454] T eb0c33c3e4014eb3b931d64adac60e1a P c5f3003b08fb4214b116c0be8dd62851 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85fbc6c98f0643469c92215a5cfb2e38 in term 2.
I20240502 07:59:09.088727 30493 raft_consensus.cc:1050] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a: attempting to promote NON_VOTER 0dfa00c061c748b18772d5f0566944b5 to VOTER
I20240502 07:59:09.089212 30493 consensus_queue.cc:237] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: false } }
I20240502 07:59:09.090922 27538 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:59:09.091413 28464 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:09.092216 25628 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 LEARNER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:59:09.092326 25497 raft_consensus.cc:1261] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Refusing update from remote peer 14d8fd1fc3a04d9d967b8033df84f34a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:59:09.093461 29320 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:09.093789 29320 consensus_queue.cc:1035] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:09.097003 28464 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: false } } }
I20240502 07:59:09.097819 25628 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P 0dfa00c061c748b18772d5f0566944b5 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: false } } }
I20240502 07:59:09.100278 25347 catalog_manager.cc:5484] T 9722dae4e4414084bdd8066b41bcc47a P 14d8fd1fc3a04d9d967b8033df84f34a reported cstate change: config changed from index 10 to 11, 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:09.100525 27538 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P c6746058199446968d2496df55cdee42 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: false } } }
I20240502 07:59:09.102761 25497 raft_consensus.cc:2932] T 9722dae4e4414084bdd8066b41bcc47a P c3be11f0fe304e0f8e008dc1e4c41f70 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 0dfa00c061c748b18772d5f0566944b5 (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "c6746058199446968d2496df55cdee42" member_type: VOTER last_known_addr { host: "127.18.13.135" port: 34615 } attrs { promote: false } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } attrs { promote: false } } }
I20240502 07:59:09.127846 27651 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:09.133442 25608 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 07:59:09.137090 25693 consensus_peers.cc:462] T e5958bb5c8fc4c1d91ae8f1e8b3861d0 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111)
I20240502 07:59:09.138958 25871 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:59:09.139096 25477 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:59:09.141003 27256 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:09.149660 27518 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:09.152015 25739 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 07:59:09.177933 27670 raft_consensus.cc:1203] T 03016ecfa11d4987a2e9d2368a66a5a2 P 85fbc6c98f0643469c92215a5cfb2e38 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240502 07:59:09.200367 25693 consensus_peers.cc:462] T 03016ecfa11d4987a2e9d2368a66a5a2 P 14d8fd1fc3a04d9d967b8033df84f34a -> Peer a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.18.13.134:43037: connect: Connection refused (error 111)
I20240502 07:59:09.201617 27720 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskeLuhys/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 07:59:02.282457 27727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:02.283450 27727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:02.288524 27727 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:59:03.283381 27727 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 07:59:04.016674 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.016937 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:04.017215 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 616a1d4f85774bf6b08fc4da5a27702a: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.017277 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.017323 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8aaedf9499044a93ab9419f10e2342e2: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.017416 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.024744 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:04.024884 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.024958 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.025015 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.025063 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 616a1d4f85774bf6b08fc4da5a27702a: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.025168 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7c7fb9171c9548a7ad2d45e295d780cb: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:04.025230 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f72d1ad6e00b46cf99f485a9ef60b9b1: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.025286 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:04.025341 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:04.025389 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server c6746058199446968d2496df55cdee42
W20240502 07:59:04.025439 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server 85fbc6c98f0643469c92215a5cfb2e38
I20240502 07:59:04.188728 27727 rebalancer_tool.cc:1074] tablet 056ba8d15d18409e946c905b8876097a: 'e9596bfb8d324b4da93a4288771ed177' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:04.270774 27727 rebalancer_tool.cc:1074] tablet ee25634bf534443b90fdbdcebdbcb02e: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> '85fbc6c98f0643469c92215a5cfb2e38' move scheduled
I20240502 07:59:04.335439 27727 rebalancer_tool.cc:1074] tablet 9722dae4e4414084bdd8066b41bcc47a: '0dfa00c061c748b18772d5f0566944b5' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:04.373966 27727 rebalancer_tool.cc:1074] tablet 5eb2de2942914800bdbe924d45a9b1e7: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'c5f3003b08fb4214b116c0be8dd62851' move scheduled
I20240502 07:59:04.491986 27727 rebalancer_tool.cc:1074] tablet 273e203d4ed44c43859f08e9bec08a9d: 'e9596bfb8d324b4da93a4288771ed177' -> '85fbc6c98f0643469c92215a5cfb2e38' move scheduled
I20240502 07:59:04.564111 27727 rebalancer_tool.cc:1074] tablet 1ad22d2e464d4defab6bd0061e6058f8: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:04.764125 27727 rebalancer_tool.cc:1074] tablet a3bc416015dc40d88743215d2cf46230: '0dfa00c061c748b18772d5f0566944b5' -> 'c5f3003b08fb4214b116c0be8dd62851' move scheduled
I20240502 07:59:04.852883 27727 rebalancer_tool.cc:1084] tablet fd68090fdadf45faa0e3b2936b30047e: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'a17bbff6a71b4667b14f5dd12bca3057' move ignored: Illegal state: tablet fd68090fdadf45faa0e3b2936b30047e: replica a17bbff6a71b4667b14f5dd12bca3057 is already present
I20240502 07:59:04.853001 27727 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:06.111434 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c6746058199446968d2496df55cdee42
W20240502 07:59:06.111544 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:06.112344 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5f087acc4d94b0f9694fb1cf95e7dca: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:06.112462 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c6746058199446968d2496df55cdee42
W20240502 07:59:06.112519 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8aaedf9499044a93ab9419f10e2342e2: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server c5f3003b08fb4214b116c0be8dd62851
W20240502 07:59:06.112558 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:06.112593 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f72d1ad6e00b46cf99f485a9ef60b9b1: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:06.112632 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table a5520c28ad46406f9fdf6855d5dfa37b: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:06.112680 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:06.112722 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 10514d1b971c4d5c9e2b1f433f40d2c1: could not find any suitable replica to move from server 0dfa00c061c748b18772d5f0566944b5 to server c6746058199446968d2496df55cdee42
W20240502 07:59:06.113592 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 7c7fb9171c9548a7ad2d45e295d780cb: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server 85fbc6c98f0643469c92215a5cfb2e38
W20240502 07:59:06.113723 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 89a9dd84e8b5432a81ef7767804381d3: could not find any suitable replica to move from server e9596bfb8d324b4da93a4288771ed177 to server a17bbff6a71b4667b14f5dd12bca3057
I20240502 07:59:06.353125 27727 rebalancer_tool.cc:1084] tablet ff31ddd6b3d74a9aae6617afb56c40f5: '14d8fd1fc3a04d9d967b8033df84f34a' -> '' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c3be11f0fe304e0f8e008dc1e4c41f70" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37377 } attrs { replace: true } } peers { permanent_uuid: "e9596bfb8d324b4da93a4288771ed177" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 45865 } } peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } attrs { replace: true } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false } }
I20240502 07:59:06.353228 27727 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:07.089025 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server 14d8fd1fc3a04d9d967b8033df84f34a
W20240502 07:59:07.089138 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 67841c5c645f4e86970122328fd1238e: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server a17bbff6a71b4667b14f5dd12bca3057
W20240502 07:59:07.089222 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 8aaedf9499044a93ab9419f10e2342e2: could not find any suitable replica to move from server 14d8fd1fc3a04d9d967b8033df84f34a to server c5f3003b08fb4214b116c0be8dd62851
I20240502 07:59:07.237695 27727 rebalancer_tool.cc:1074] tablet 87d6266008784883891ca32c01363dbe: '14d8fd1fc3a04d9d967b8033df84f34a' -> '' move scheduled
I20240502 07:59:07.328210 27727 rebalancer_tool.cc:1074] tablet a1fa713b432740d98bcab918d251b54d: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'a17bbff6a71b4667b14f5dd12bca3057' move scheduled
I20240502 07:59:07.354437 27727 rebalancer_tool.cc:1074] tablet 8904fa47fc274c14acd291b4c91964be: 'e9596bfb8d324b4da93a4288771ed177' -> '' move scheduled
I20240502 07:59:07.391377 27727 rebalancer_tool.cc:1084] tablet dbfd23aa383f413cbe4a5ccd8146d2b6: '0dfa00c061c748b18772d5f0566944b5' -> 'c5f3003b08fb4214b116c0be8dd62851' move ignored: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240502 07:59:07.391458 27727 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 07:59:07.758270 27727 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9eeb74ff004488ba3049c0e8e535bb5: could not find any suitable replica to move from server c3be11f0fe304e0f8e008dc1e4c41f70 to server 14d8fd1fc3a04d9d967b8033df84f34a
I20240502 07:59:07.789464 27727 rebalancer_tool.cc:1074] tablet 71a67cb0f6b54bfb9eb44400fe21ce40: '14d8fd1fc3a04d9d967b8033df84f34a' -> '0dfa00c061c748b18772d5f0566944b5' move scheduled
I20240502 07:59:07.989794 27727 rebalancer_tool.cc:1074] tablet 0aa60608089c4f50bcee339d01815a65: 'a17bbff6a71b4667b14f5dd12bca3057' -> 'c5f3003b08fb4214b116c0be8dd62851' move scheduled
I20240502 07:59:08.097748 27727 rebalancer_tool.cc:1074] tablet 03016ecfa11d4987a2e9d2368a66a5a2: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> '85fbc6c98f0643469c92215a5cfb2e38' move scheduled
I20240502 07:59:08.411145 27727 rebalancer_tool.cc:1074] tablet 99c3ddedadbf4c0e8858085b66022342: '0dfa00c061c748b18772d5f0566944b5' -> 'c6746058199446968d2496df55cdee42' move scheduled
I20240502 07:59:08.525672 27727 rebalancer_tool.cc:1074] tablet eb0c33c3e4014eb3b931d64adac60e1a: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> '' move scheduled
I20240502 07:59:08.731140 27727 rebalancer_tool.cc:1074] tablet fc1ac05112724e0ea1f84f648598304e: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'e9596bfb8d324b4da93a4288771ed177' move scheduled
I20240502 07:59:08.772403 27727 rebalancer_tool.cc:1074] tablet d8e50f3d37d842868900a4cff5bf3601: 'a17bbff6a71b4667b14f5dd12bca3057' -> 'c5f3003b08fb4214b116c0be8dd62851' move scheduled
I20240502 07:59:08.804210 27727 rebalancer_tool.cc:1074] tablet f21d6014b91c45df96325017b57f19e3: 'a17bbff6a71b4667b14f5dd12bca3057' -> '85fbc6c98f0643469c92215a5cfb2e38' move scheduled
I20240502 07:59:08.931236 27727 rebalancer_tool.cc:1136] tablet 5eb2de2942914800bdbe924d45a9b1e7: '14d8fd1fc3a04d9d967b8033df84f34a' -> 'c5f3003b08fb4214b116c0be8dd62851' move completed: OK
I20240502 07:59:08.942521 27727 rebalancer_tool.cc:1136] tablet 056ba8d15d18409e946c905b8876097a: 'e9596bfb8d324b4da93a4288771ed177' -> 'c6746058199446968d2496df55cdee42' move completed: OK
I20240502 07:59:08.947932 27727 rebalancer_tool.cc:1136] tablet 9722dae4e4414084bdd8066b41bcc47a: '0dfa00c061c748b18772d5f0566944b5' -> 'a17bbff6a71b4667b14f5dd12bca3057' move completed: Incomplete: tablet 9722dae4e4414084bdd8066b41bcc47a, TS 0dfa00c061c748b18772d5f0566944b5 -> TS a17bbff6a71b4667b14f5dd12bca3057 move failed, target replica disappeared
I20240502 07:59:08.953802 27727 rebalancer_tool.cc:1136] tablet a3bc416015dc40d88743215d2cf46230: '0dfa00c061c748b18772d5f0566944b5' -> 'c5f3003b08fb4214b116c0be8dd62851' move completed: OK
I20240502 07:59:08.964637 27727 rebalancer_tool.cc:1136] tablet 8904fa47fc274c14acd291b4c91964be: 'e9596bfb8d324b4da93a4288771ed177' -> '' move completed: OK
I20240502 07:59:08.980101 27727 rebalancer_tool.cc:1136] tablet 273e203d4ed44c43859f08e9bec08a9d: 'e9596bfb8d324b4da93a4288771ed177' -> '85fbc6c98f0643469c92215a5cfb2e38' move completed: OK
I20240502 07:59:09.002192 27727 rebalancer_tool.cc:1136] tablet 87d6266008784883891ca32c01363dbe: '14d8fd1fc3a04d9d967b8033df84f34a' -> '' move completed: OK
I20240502 07:59:09.010130 27727 rebalancer_tool.cc:1136] tablet 1ad22d2e464d4defab6bd0061e6058f8: 'c3be11f0fe304e0f8e008dc1e4c41f70' -> 'c6746058199446968d2496df55cdee42' move completed: Incomplete: tablet 1ad22d2e464d4defab6bd0061e6058f8, TS c3be11f0fe304e0f8e008dc1e4c41f70 -> TS c6746058199446968d2496df55cdee42 move failed, target replica disappeared
I20240502 07:59:09.039583 27727 rebalancer_tool.cc:1084] tablet 843d1c1d6384476983749a78f65a5c74: '0dfa00c061c748b18772d5f0566944b5' -> 'c3be11f0fe304e0f8e008dc1e4c41f70' move ignored: Not found: Peer with uuid 0dfa00c061c748b18772d5f0566944b5 not found in consensus config
I20240502 07:59:09.039685 27727 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server a17bbff6a71b4667b14f5dd12bca3057 (127.18.13.134:43037): unacceptable health status UNAVAILABLE
E20240502 07:59:09.201907 18486 external_mini_cluster.cc:943] Process with UUID a17bbff6a71b4667b14f5dd12bca3057 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 1
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240502 07:59:09.202600 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 25416
I20240502 07:59:09.211050 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 25547
I20240502 07:59:09.216794 27275 raft_consensus.cc:2932] T 5a0605c8c64f4c6fb5e7315a118a0d7f P c5f3003b08fb4214b116c0be8dd62851 [term 2 LEARNER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "14d8fd1fc3a04d9d967b8033df84f34a" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 42205 } } peers { permanent_uuid: "0dfa00c061c748b18772d5f0566944b5" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 38825 } } peers { permanent_uuid: "a17bbff6a71b4667b14f5dd12bca3057" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 43037 } attrs { promote: false replace: true } } peers { permanent_uuid: "c5f3003b08fb4214b116c0be8dd62851" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 39097 } attrs { promote: true } } }
I20240502 07:59:09.219386 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 25678
I20240502 07:59:09.227737 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 25809
I20240502 07:59:09.235494 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 27194
W20240502 07:59:09.238634 27607 consensus_peers.cc:485] T 8904fa47fc274c14acd291b4c91964be P 85fbc6c98f0643469c92215a5cfb2e38 -> Peer c3be11f0fe304e0f8e008dc1e4c41f70 (127.18.13.129:37377): Couldn't send request to peer c3be11f0fe304e0f8e008dc1e4c41f70. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.129:37377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:09.242236 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 27457
I20240502 07:59:09.248755 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 27588
I20240502 07:59:09.255092 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 25325
2024-05-02T07:59:09Z chronyd exiting
I20240502 07:59:09.273345 18486 test_util.cc:175] -----------------------------------------------
I20240502 07:59:09.273411 18486 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714636688619157-18486-0
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00> (19467 ms)
[----------] 1 test from ConcurrentRebalancersTest (19467 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1
I20240502 07:59:09.274104 18486 test_util.cc:268] Using random seed: -94635021
I20240502 07:59:09.274876 18486 ts_itest-base.cc:112] Starting cluster with:
I20240502 07:59:09.274919 18486 ts_itest-base.cc:113] --------------
I20240502 07:59:09.274930 18486 ts_itest-base.cc:114] 4 tablet servers
I20240502 07:59:09.274940 18486 ts_itest-base.cc:115] 1 replicas per TS
I20240502 07:59:09.274953 18486 ts_itest-base.cc:116] --------------
2024-05-02T07:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T07:59:09Z Disabled control of system clock
I20240502 07:59:09.283607 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.13.190:34323
--webserver_interface=127.18.13.190
--webserver_port=0
--builtin_ntp_servers=127.18.13.148:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.13.190:34323
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240502 07:59:09.392036 30654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:09.392287 30654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:09.392345 30654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:09.395445 30654 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:09.395520 30654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 07:59:09.395567 30654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:09.395628 30654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 07:59:09.395675 30654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 07:59:09.399547 30654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.18.13.190:34323
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.13.190:34323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.18.13.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.30654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:09.400542 30654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:09.401710 30654 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:09.408792 30662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:09.409454 30659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:09.409994 30660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:09.410835 30654 server_base.cc:1034] running on GCE node
I20240502 07:59:09.411357 30654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:09.412350 30654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:09.413501 30654 hybrid_clock.cc:648] HybridClock initialized: now 1714636749413480 us; error 39 us; skew 500 ppm
I20240502 07:59:09.415452 30654 webserver.cc:460] Webserver started at http://127.18.13.190:39223/ using document root <none> and password file <none>
I20240502 07:59:09.416059 30654 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:09.416121 30654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:09.416312 30654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:09.418020 30654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "53d352a2e03b4fce85db44fac169c811"
format_stamp: "Formatted at 2024-05-02 07:59:09 on dist-test-slave-f46z"
I20240502 07:59:09.418529 30654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "53d352a2e03b4fce85db44fac169c811"
format_stamp: "Formatted at 2024-05-02 07:59:09 on dist-test-slave-f46z"
I20240502 07:59:09.421913 30654 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:09.424082 30668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:09.425043 30654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:09.425151 30654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
uuid: "53d352a2e03b4fce85db44fac169c811"
format_stamp: "Formatted at 2024-05-02 07:59:09 on dist-test-slave-f46z"
I20240502 07:59:09.425235 30654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:09.439102 30654 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:09.439767 30654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:09.439970 30654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:09.446419 30654 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.190:34323
I20240502 07:59:09.446429 30720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.190:34323 every 8 connection(s)
I20240502 07:59:09.447388 30654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 07:59:09.449137 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 30654
I20240502 07:59:09.449280 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 07:59:09.450019 30721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:09.456043 30721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811: Bootstrap starting.
I20240502 07:59:09.458444 30721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:09.459234 30721 log.cc:824] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:09.460857 30721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811: No bootstrap required, opened a new log
I20240502 07:59:09.463407 30721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d352a2e03b4fce85db44fac169c811" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 34323 } }
I20240502 07:59:09.463598 30721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:09.463673 30721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d352a2e03b4fce85db44fac169c811, State: Initialized, Role: FOLLOWER
I20240502 07:59:09.464103 30721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d352a2e03b4fce85db44fac169c811" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 34323 } }
I20240502 07:59:09.464237 30721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:09.464298 30721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:09.464407 30721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:09.465147 30721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d352a2e03b4fce85db44fac169c811" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 34323 } }
I20240502 07:59:09.465462 30721 leader_election.cc:302] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53d352a2e03b4fce85db44fac169c811; no voters:
I20240502 07:59:09.465721 30721 leader_election.cc:288] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:09.466095 30726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:09.466295 30726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [term 1 LEADER]: Becoming Leader. State: Replica: 53d352a2e03b4fce85db44fac169c811, State: Running, Role: LEADER
I20240502 07:59:09.466580 30726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d352a2e03b4fce85db44fac169c811" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 34323 } }
I20240502 07:59:09.466732 30721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:59:09.468225 30728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53d352a2e03b4fce85db44fac169c811. Latest consensus state: current_term: 1 leader_uuid: "53d352a2e03b4fce85db44fac169c811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d352a2e03b4fce85db44fac169c811" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 34323 } } }
I20240502 07:59:09.468273 30727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53d352a2e03b4fce85db44fac169c811" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d352a2e03b4fce85db44fac169c811" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 34323 } } }
I20240502 07:59:09.468355 30728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:09.468369 30727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811 [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:09.468760 30738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:59:09.471408 30738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:59:09.476083 30738 catalog_manager.cc:1347] Generated new cluster ID: c5cf55ccc0c845cba7215269577334ba
I20240502 07:59:09.476159 30738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:59:09.489665 30738 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:59:09.490947 30738 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:59:09.501857 30738 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 53d352a2e03b4fce85db44fac169c811: Generated new TSK 0
I20240502 07:59:09.502609 30738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:59:09.514572 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.129:0
--local_ip_for_outbound_sockets=127.18.13.129
--webserver_interface=127.18.13.129
--webserver_port=0
--tserver_master_addrs=127.18.13.190:34323
--builtin_ntp_servers=127.18.13.148:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 07:59:09.623097 30745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:09.623333 30745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:09.623376 30745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:09.626453 30745 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:09.626523 30745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:09.626592 30745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.129
I20240502 07:59:09.630537 30745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.18.13.129
--webserver_port=0
--tserver_master_addrs=127.18.13.190:34323
--never_fsync=true
--heap_profile_path=/tmp/kudu.30745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.129
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:09.631521 30745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:09.632690 30745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:09.640717 30751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:09.641316 30750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:09.641899 30753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:09.642746 30745 server_base.cc:1034] running on GCE node
I20240502 07:59:09.643141 30745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:09.643741 30745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:09.644935 30745 hybrid_clock.cc:648] HybridClock initialized: now 1714636749644922 us; error 28 us; skew 500 ppm
I20240502 07:59:09.646719 30745 webserver.cc:460] Webserver started at http://127.18.13.129:43849/ using document root <none> and password file <none>
I20240502 07:59:09.647279 30745 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:09.647344 30745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:09.647527 30745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:09.649211 30745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "050007925f4f41788b9a6bc228db41b0"
format_stamp: "Formatted at 2024-05-02 07:59:09 on dist-test-slave-f46z"
I20240502 07:59:09.649675 30745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "050007925f4f41788b9a6bc228db41b0"
format_stamp: "Formatted at 2024-05-02 07:59:09 on dist-test-slave-f46z"
I20240502 07:59:09.652918 30745 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 07:59:09.654996 30759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:09.656034 30745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:09.656144 30745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "050007925f4f41788b9a6bc228db41b0"
format_stamp: "Formatted at 2024-05-02 07:59:09 on dist-test-slave-f46z"
I20240502 07:59:09.656226 30745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:09.667591 30745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:09.668236 30745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:09.668398 30745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:09.668954 30745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:09.669795 30745 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:09.669843 30745 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:09.669903 30745 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:09.669931 30745 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:09.678488 30745 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.129:40133
I20240502 07:59:09.678507 30872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.129:40133 every 8 connection(s)
I20240502 07:59:09.679452 30745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 07:59:09.680373 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 30745
I20240502 07:59:09.680501 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 07:59:09.682768 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.130:0
--local_ip_for_outbound_sockets=127.18.13.130
--webserver_interface=127.18.13.130
--webserver_port=0
--tserver_master_addrs=127.18.13.190:34323
--builtin_ntp_servers=127.18.13.148:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:09.689256 30873 heartbeater.cc:347] Connected to a master server at 127.18.13.190:34323
I20240502 07:59:09.689482 30873 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:09.690017 30873 heartbeater.cc:511] Master 127.18.13.190:34323 requested a full tablet report, sending...
I20240502 07:59:09.691469 30685 ts_manager.cc:194] Registered new tserver with Master: 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133)
I20240502 07:59:09.692695 30685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.129:38407
W20240502 07:59:09.793433 30876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:09.793666 30876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:09.793712 30876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:09.796778 30876 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:09.796844 30876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:09.796988 30876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.130
I20240502 07:59:09.800578 30876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.18.13.130
--webserver_port=0
--tserver_master_addrs=127.18.13.190:34323
--never_fsync=true
--heap_profile_path=/tmp/kudu.30876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.130
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:09.801488 30876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:09.802644 30876 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:09.811008 30884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:09.811178 30881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:09.811810 30882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:09.812474 30876 server_base.cc:1034] running on GCE node
I20240502 07:59:09.812882 30876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:09.813513 30876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:09.814667 30876 hybrid_clock.cc:648] HybridClock initialized: now 1714636749814642 us; error 43 us; skew 500 ppm
I20240502 07:59:09.816473 30876 webserver.cc:460] Webserver started at http://127.18.13.130:33993/ using document root <none> and password file <none>
I20240502 07:59:09.817054 30876 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:09.817116 30876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:09.817291 30876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:09.818969 30876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9f1fe3e3d9d34dd88971628d6018d833"
format_stamp: "Formatted at 2024-05-02 07:59:09 on dist-test-slave-f46z"
I20240502 07:59:09.819453 30876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9f1fe3e3d9d34dd88971628d6018d833"
format_stamp: "Formatted at 2024-05-02 07:59:09 on dist-test-slave-f46z"
I20240502 07:59:09.822796 30876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 07:59:09.824993 30890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:09.825987 30876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:09.826102 30876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9f1fe3e3d9d34dd88971628d6018d833"
format_stamp: "Formatted at 2024-05-02 07:59:09 on dist-test-slave-f46z"
I20240502 07:59:09.826189 30876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:09.867738 30876 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:09.868412 30876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:09.868592 30876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:09.869144 30876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:09.870045 30876 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:09.870096 30876 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:09.870155 30876 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:09.870187 30876 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:09.878609 30876 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.130:40385
I20240502 07:59:09.878645 31003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.130:40385 every 8 connection(s)
I20240502 07:59:09.879549 30876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 07:59:09.888103 31004 heartbeater.cc:347] Connected to a master server at 127.18.13.190:34323
I20240502 07:59:09.888334 31004 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:09.888829 31004 heartbeater.cc:511] Master 127.18.13.190:34323 requested a full tablet report, sending...
I20240502 07:59:09.889490 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 30876
I20240502 07:59:09.889607 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 07:59:09.889809 30685 ts_manager.cc:194] Registered new tserver with Master: 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130:40385)
I20240502 07:59:09.890446 30685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.130:49483
I20240502 07:59:09.891942 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.131:0
--local_ip_for_outbound_sockets=127.18.13.131
--webserver_interface=127.18.13.131
--webserver_port=0
--tserver_master_addrs=127.18.13.190:34323
--builtin_ntp_servers=127.18.13.148:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 07:59:09.999555 31007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:09.999838 31007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:09.999918 31007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:10.002988 31007 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:10.003062 31007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:10.003165 31007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.131
I20240502 07:59:10.006695 31007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.18.13.131
--webserver_port=0
--tserver_master_addrs=127.18.13.190:34323
--never_fsync=true
--heap_profile_path=/tmp/kudu.31007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.131
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:10.007619 31007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:10.008790 31007 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:10.016705 31012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:10.017370 31015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:10.017956 31013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:10.018786 31007 server_base.cc:1034] running on GCE node
I20240502 07:59:10.019157 31007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:10.019816 31007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:10.020972 31007 hybrid_clock.cc:648] HybridClock initialized: now 1714636750020948 us; error 41 us; skew 500 ppm
I20240502 07:59:10.022763 31007 webserver.cc:460] Webserver started at http://127.18.13.131:45687/ using document root <none> and password file <none>
I20240502 07:59:10.023321 31007 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:10.023383 31007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:10.023555 31007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:10.025246 31007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c6642c4ab0104a9da2b204f9e965abfd"
format_stamp: "Formatted at 2024-05-02 07:59:10 on dist-test-slave-f46z"
I20240502 07:59:10.025722 31007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c6642c4ab0104a9da2b204f9e965abfd"
format_stamp: "Formatted at 2024-05-02 07:59:10 on dist-test-slave-f46z"
I20240502 07:59:10.029062 31007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 07:59:10.031345 31021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:10.032377 31007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 07:59:10.032487 31007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c6642c4ab0104a9da2b204f9e965abfd"
format_stamp: "Formatted at 2024-05-02 07:59:10 on dist-test-slave-f46z"
I20240502 07:59:10.032573 31007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:10.054921 31007 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:10.055560 31007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:10.055725 31007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:10.056267 31007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:10.057134 31007 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:10.057184 31007 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:10.057246 31007 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:10.057283 31007 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:10.065472 31007 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.131:38213
I20240502 07:59:10.065495 31134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.131:38213 every 8 connection(s)
I20240502 07:59:10.066404 31007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 07:59:10.067693 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 31007
I20240502 07:59:10.067849 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 07:59:10.069983 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.132:0
--local_ip_for_outbound_sockets=127.18.13.132
--webserver_interface=127.18.13.132
--webserver_port=0
--tserver_master_addrs=127.18.13.190:34323
--builtin_ntp_servers=127.18.13.148:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:10.075592 31135 heartbeater.cc:347] Connected to a master server at 127.18.13.190:34323
I20240502 07:59:10.075852 31135 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:10.076398 31135 heartbeater.cc:511] Master 127.18.13.190:34323 requested a full tablet report, sending...
I20240502 07:59:10.077447 30684 ts_manager.cc:194] Registered new tserver with Master: c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131:38213)
I20240502 07:59:10.078255 30684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.131:36195
W20240502 07:59:10.180462 31138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:10.180729 31138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:10.180811 31138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:10.183815 31138 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:10.183889 31138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:10.183987 31138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.132
I20240502 07:59:10.187594 31138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.18.13.132
--webserver_port=0
--tserver_master_addrs=127.18.13.190:34323
--never_fsync=true
--heap_profile_path=/tmp/kudu.31138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.132
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:10.188549 31138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:10.189669 31138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:10.198491 31146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:10.198634 31143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:10.199290 31144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:10.200001 31138 server_base.cc:1034] running on GCE node
I20240502 07:59:10.200423 31138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:10.201086 31138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:10.202248 31138 hybrid_clock.cc:648] HybridClock initialized: now 1714636750202233 us; error 28 us; skew 500 ppm
I20240502 07:59:10.204150 31138 webserver.cc:460] Webserver started at http://127.18.13.132:46147/ using document root <none> and password file <none>
I20240502 07:59:10.204718 31138 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:10.204783 31138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:10.204960 31138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:10.206642 31138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "9b205acd80ed40cdad98e381fb296499"
format_stamp: "Formatted at 2024-05-02 07:59:10 on dist-test-slave-f46z"
I20240502 07:59:10.207124 31138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "9b205acd80ed40cdad98e381fb296499"
format_stamp: "Formatted at 2024-05-02 07:59:10 on dist-test-slave-f46z"
I20240502 07:59:10.210633 31138 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20240502 07:59:10.212839 31152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:10.213842 31138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:10.213948 31138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "9b205acd80ed40cdad98e381fb296499"
format_stamp: "Formatted at 2024-05-02 07:59:10 on dist-test-slave-f46z"
I20240502 07:59:10.214044 31138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:10.230304 31138 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:10.230976 31138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:10.231134 31138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:10.231673 31138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:10.232599 31138 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:10.232650 31138 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:10.232697 31138 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:10.232728 31138 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:10.241219 31138 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.132:38863
I20240502 07:59:10.241246 31265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.132:38863 every 8 connection(s)
I20240502 07:59:10.242190 31138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:59:10.245981 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 31138
I20240502 07:59:10.246111 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 07:59:10.251472 31266 heartbeater.cc:347] Connected to a master server at 127.18.13.190:34323
I20240502 07:59:10.251685 31266 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:10.252250 31266 heartbeater.cc:511] Master 127.18.13.190:34323 requested a full tablet report, sending...
I20240502 07:59:10.253175 30684 ts_manager.cc:194] Registered new tserver with Master: 9b205acd80ed40cdad98e381fb296499 (127.18.13.132:38863)
I20240502 07:59:10.253715 30684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.132:39541
I20240502 07:59:10.260221 18486 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 07:59:10.268975 30684 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49212:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 07:59:10.284571 30938 tablet_service.cc:1467] Processing CreateTablet for tablet 584032d120a34aa4821240f2eeb7e96c (DEFAULT_TABLE table=TestTable [id=6ed1b623b2ff4addbff0cba22a648c38]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:10.285444 30938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584032d120a34aa4821240f2eeb7e96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:10.290231 31279 tablet_bootstrap.cc:492] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833: Bootstrap starting.
I20240502 07:59:10.292204 31279 tablet_bootstrap.cc:654] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:10.292882 31279 log.cc:824] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:10.294406 31279 tablet_bootstrap.cc:492] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833: No bootstrap required, opened a new log
I20240502 07:59:10.294590 31279 ts_tablet_manager.cc:1367] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 07:59:10.297587 31279 raft_consensus.cc:348] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:10.297770 31279 raft_consensus.cc:374] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:10.297809 31279 raft_consensus.cc:729] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Initialized, Role: FOLLOWER
I20240502 07:59:10.298249 31279 consensus_queue.cc:260] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:10.298385 31279 raft_consensus.cc:388] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:10.298449 31279 raft_consensus.cc:482] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:10.298556 31279 raft_consensus.cc:3037] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:10.299367 31279 raft_consensus.cc:504] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:10.299680 31279 leader_election.cc:302] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f1fe3e3d9d34dd88971628d6018d833; no voters:
I20240502 07:59:10.299983 31279 leader_election.cc:288] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:10.300109 31281 raft_consensus.cc:2781] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:10.300310 31281 raft_consensus.cc:686] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Becoming Leader. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Running, Role: LEADER
I20240502 07:59:10.300609 31281 consensus_queue.cc:237] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:10.300716 31004 heartbeater.cc:503] Master 127.18.13.190:34323 was elected leader, sending a full tablet report...
I20240502 07:59:10.300839 31279 ts_tablet_manager.cc:1398] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 07:59:10.303975 30684 catalog_manager.cc:5484] T 584032d120a34aa4821240f2eeb7e96c P 9f1fe3e3d9d34dd88971628d6018d833 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "9f1fe3e3d9d34dd88971628d6018d833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:10.311139 18486 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 07:59:10.312506 18486 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 050007925f4f41788b9a6bc228db41b0 to finish bootstrapping
I20240502 07:59:10.318281 18486 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9f1fe3e3d9d34dd88971628d6018d833 to finish bootstrapping
I20240502 07:59:10.323252 18486 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c6642c4ab0104a9da2b204f9e965abfd to finish bootstrapping
I20240502 07:59:10.328711 18486 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9b205acd80ed40cdad98e381fb296499 to finish bootstrapping
I20240502 07:59:10.334331 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 31007
I20240502 07:59:10.339967 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 31138
I20240502 07:59:10.694976 30873 heartbeater.cc:503] Master 127.18.13.190:34323 was elected leader, sending a full tablet report...
I20240502 07:59:14.096527 30684 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49212:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:14.108042 30806 tablet_service.cc:1467] Processing CreateTablet for tablet 39209b4f0a1444b6b80a155657217fe8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ead4dfe17a8246e8b14dd665edd372ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:14.108057 30805 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3a9771d3a9438d9dbff501500386ef (DEFAULT_TABLE table=rebalance_test_table_0 [id=ead4dfe17a8246e8b14dd665edd372ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:14.109108 30805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3a9771d3a9438d9dbff501500386ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:14.109464 30806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39209b4f0a1444b6b80a155657217fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:14.108058 30807 tablet_service.cc:1467] Processing CreateTablet for tablet 3741c7c8018b412da5fd73fbbf052102 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ead4dfe17a8246e8b14dd665edd372ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:14.111263 30807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3741c7c8018b412da5fd73fbbf052102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:14.115597 31296 tablet_bootstrap.cc:492] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0: Bootstrap starting.
I20240502 07:59:14.117502 31296 tablet_bootstrap.cc:654] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:14.118461 31296 log.cc:824] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:14.120018 31296 tablet_bootstrap.cc:492] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0: No bootstrap required, opened a new log
I20240502 07:59:14.120204 31296 ts_tablet_manager.cc:1367] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 07:59:14.122671 31296 raft_consensus.cc:348] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.122848 31296 raft_consensus.cc:374] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:14.122884 31296 raft_consensus.cc:729] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Initialized, Role: FOLLOWER
I20240502 07:59:14.123330 31296 consensus_queue.cc:260] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.123448 31296 raft_consensus.cc:388] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:14.123512 31296 raft_consensus.cc:482] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:14.123617 31296 raft_consensus.cc:3037] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:14.124575 31296 raft_consensus.cc:504] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.124913 31296 leader_election.cc:302] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 050007925f4f41788b9a6bc228db41b0; no voters:
I20240502 07:59:14.125255 31296 leader_election.cc:288] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:14.125372 31298 raft_consensus.cc:2781] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:14.125591 31298 raft_consensus.cc:686] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Becoming Leader. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Running, Role: LEADER
I20240502 07:59:14.126047 31296 ts_tablet_manager.cc:1398] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 07:59:14.126403 31296 tablet_bootstrap.cc:492] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0: Bootstrap starting.
I20240502 07:59:14.127202 31296 tablet_bootstrap.cc:654] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:14.125890 31298 consensus_queue.cc:237] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.128284 31296 tablet_bootstrap.cc:492] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0: No bootstrap required, opened a new log
I20240502 07:59:14.128401 31296 ts_tablet_manager.cc:1367] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:14.128932 31296 raft_consensus.cc:348] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.129050 31296 raft_consensus.cc:374] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:14.129089 31296 raft_consensus.cc:729] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Initialized, Role: FOLLOWER
I20240502 07:59:14.129246 31296 consensus_queue.cc:260] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.129324 31296 raft_consensus.cc:388] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:14.129361 31296 raft_consensus.cc:482] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:14.129403 31296 raft_consensus.cc:3037] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:14.130342 31296 raft_consensus.cc:504] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.130450 31296 leader_election.cc:302] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 050007925f4f41788b9a6bc228db41b0; no voters:
I20240502 07:59:14.130542 31296 leader_election.cc:288] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:14.130717 31298 raft_consensus.cc:2781] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:14.130815 31298 raft_consensus.cc:686] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Becoming Leader. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Running, Role: LEADER
I20240502 07:59:14.130841 31296 ts_tablet_manager.cc:1398] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:14.130940 31298 consensus_queue.cc:237] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.131009 31296 tablet_bootstrap.cc:492] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0: Bootstrap starting.
I20240502 07:59:14.131944 31296 tablet_bootstrap.cc:654] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:14.133080 31296 tablet_bootstrap.cc:492] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0: No bootstrap required, opened a new log
I20240502 07:59:14.133164 31296 ts_tablet_manager.cc:1367] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:14.133764 31296 raft_consensus.cc:348] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.133852 31296 raft_consensus.cc:374] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:14.133886 31296 raft_consensus.cc:729] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Initialized, Role: FOLLOWER
I20240502 07:59:14.134007 31296 consensus_queue.cc:260] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.134088 31296 raft_consensus.cc:388] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:14.134135 31296 raft_consensus.cc:482] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:14.134186 31296 raft_consensus.cc:3037] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:14.135031 31296 raft_consensus.cc:504] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.135149 31296 leader_election.cc:302] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 050007925f4f41788b9a6bc228db41b0; no voters:
I20240502 07:59:14.135123 30684 catalog_manager.cc:5484] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 050007925f4f41788b9a6bc228db41b0 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:14.135246 31296 leader_election.cc:288] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:14.135331 31299 raft_consensus.cc:2781] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:14.135444 31299 raft_consensus.cc:686] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Becoming Leader. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Running, Role: LEADER
I20240502 07:59:14.135591 31296 ts_tablet_manager.cc:1398] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:14.135624 31299 consensus_queue.cc:237] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.139271 30684 catalog_manager.cc:5484] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 050007925f4f41788b9a6bc228db41b0 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:14.139516 30684 catalog_manager.cc:5484] T 5b3a9771d3a9438d9dbff501500386ef P 050007925f4f41788b9a6bc228db41b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 050007925f4f41788b9a6bc228db41b0 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:14.306398 30684 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49212:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:14.312017 30938 tablet_service.cc:1467] Processing CreateTablet for tablet 4d43114a3f804c84b451505f967abe5c (DEFAULT_TABLE table=rebalance_test_table_1 [id=b64ba5c7c71945868fde0618a6d5f280]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:14.312021 30937 tablet_service.cc:1467] Processing CreateTablet for tablet c25e35fdef7649a0abd4d95259c11577 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b64ba5c7c71945868fde0618a6d5f280]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:14.312026 30936 tablet_service.cc:1467] Processing CreateTablet for tablet aa53770eb24c49938a6b0f56bdb85d24 (DEFAULT_TABLE table=rebalance_test_table_1 [id=b64ba5c7c71945868fde0618a6d5f280]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:14.312417 30938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d43114a3f804c84b451505f967abe5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:14.312608 30937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25e35fdef7649a0abd4d95259c11577. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:14.312779 30936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa53770eb24c49938a6b0f56bdb85d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:14.314713 31326 tablet_bootstrap.cc:492] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833: Bootstrap starting.
I20240502 07:59:14.315538 31326 tablet_bootstrap.cc:654] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:14.316561 31326 tablet_bootstrap.cc:492] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833: No bootstrap required, opened a new log
I20240502 07:59:14.316642 31326 ts_tablet_manager.cc:1367] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:14.317137 31326 raft_consensus.cc:348] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.317229 31326 raft_consensus.cc:374] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:14.317263 31326 raft_consensus.cc:729] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Initialized, Role: FOLLOWER
I20240502 07:59:14.317411 31326 consensus_queue.cc:260] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.317492 31326 raft_consensus.cc:388] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:14.317539 31326 raft_consensus.cc:482] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:14.317587 31326 raft_consensus.cc:3037] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:14.318370 31326 raft_consensus.cc:504] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.318491 31326 leader_election.cc:302] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f1fe3e3d9d34dd88971628d6018d833; no voters:
I20240502 07:59:14.318646 31326 leader_election.cc:288] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:14.318737 31328 raft_consensus.cc:2781] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:14.318918 31326 ts_tablet_manager.cc:1398] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:14.318955 31328 raft_consensus.cc:686] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Becoming Leader. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Running, Role: LEADER
I20240502 07:59:14.319062 31326 tablet_bootstrap.cc:492] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833: Bootstrap starting.
I20240502 07:59:14.319074 31328 consensus_queue.cc:237] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.320025 31326 tablet_bootstrap.cc:654] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:14.320421 30684 catalog_manager.cc:5484] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "9f1fe3e3d9d34dd88971628d6018d833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:14.320959 31326 tablet_bootstrap.cc:492] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833: No bootstrap required, opened a new log
I20240502 07:59:14.321040 31326 ts_tablet_manager.cc:1367] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:14.322032 31326 raft_consensus.cc:348] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.322121 31326 raft_consensus.cc:374] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:14.322154 31326 raft_consensus.cc:729] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Initialized, Role: FOLLOWER
I20240502 07:59:14.322253 31326 consensus_queue.cc:260] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.322315 31326 raft_consensus.cc:388] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:14.322337 31326 raft_consensus.cc:482] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:14.322363 31326 raft_consensus.cc:3037] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:14.323091 31326 raft_consensus.cc:504] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.323191 31326 leader_election.cc:302] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f1fe3e3d9d34dd88971628d6018d833; no voters:
I20240502 07:59:14.323293 31326 leader_election.cc:288] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:14.323339 31328 raft_consensus.cc:2781] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:14.323438 31328 raft_consensus.cc:686] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Becoming Leader. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Running, Role: LEADER
I20240502 07:59:14.323599 31326 ts_tablet_manager.cc:1398] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:14.323560 31328 consensus_queue.cc:237] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.323767 31326 tablet_bootstrap.cc:492] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833: Bootstrap starting.
I20240502 07:59:14.324652 31326 tablet_bootstrap.cc:654] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:14.324921 30684 catalog_manager.cc:5484] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "9f1fe3e3d9d34dd88971628d6018d833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:14.325551 31326 tablet_bootstrap.cc:492] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833: No bootstrap required, opened a new log
I20240502 07:59:14.325630 31326 ts_tablet_manager.cc:1367] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:14.326026 31326 raft_consensus.cc:348] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.326107 31326 raft_consensus.cc:374] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:14.326144 31326 raft_consensus.cc:729] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Initialized, Role: FOLLOWER
I20240502 07:59:14.326251 31326 consensus_queue.cc:260] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.326324 31326 raft_consensus.cc:388] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:14.326346 31326 raft_consensus.cc:482] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:14.326373 31326 raft_consensus.cc:3037] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:14.327029 31326 raft_consensus.cc:504] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.327118 31326 leader_election.cc:302] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f1fe3e3d9d34dd88971628d6018d833; no voters:
I20240502 07:59:14.327181 31326 leader_election.cc:288] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:14.327231 31328 raft_consensus.cc:2781] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:14.327324 31326 ts_tablet_manager.cc:1398] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 07:59:14.327487 31328 raft_consensus.cc:686] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Becoming Leader. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Running, Role: LEADER
I20240502 07:59:14.327610 31328 consensus_queue.cc:237] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.328748 30684 catalog_manager.cc:5484] T c25e35fdef7649a0abd4d95259c11577 P 9f1fe3e3d9d34dd88971628d6018d833 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "9f1fe3e3d9d34dd88971628d6018d833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:14.493921 30684 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49212:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:14.500288 30806 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9fbec1717044a5839d0d0f2c967f6f (DEFAULT_TABLE table=rebalance_test_table_2 [id=fa7ed001f0f5412481ea3f02e7239eb3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:14.500344 30807 tablet_service.cc:1467] Processing CreateTablet for tablet de57879c802a4ac08fa6d96efed7360a (DEFAULT_TABLE table=rebalance_test_table_2 [id=fa7ed001f0f5412481ea3f02e7239eb3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:14.500591 30806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9fbec1717044a5839d0d0f2c967f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:14.501116 30805 tablet_service.cc:1467] Processing CreateTablet for tablet 6fdbdb698ee64caf9e478368aba84e60 (DEFAULT_TABLE table=rebalance_test_table_2 [id=fa7ed001f0f5412481ea3f02e7239eb3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:14.501379 30805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fdbdb698ee64caf9e478368aba84e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:14.502427 31296 tablet_bootstrap.cc:492] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0: Bootstrap starting.
I20240502 07:59:14.503356 30807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de57879c802a4ac08fa6d96efed7360a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:14.503389 31296 tablet_bootstrap.cc:654] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:14.504516 31296 tablet_bootstrap.cc:492] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0: No bootstrap required, opened a new log
I20240502 07:59:14.504602 31296 ts_tablet_manager.cc:1367] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:14.505082 31296 raft_consensus.cc:348] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.505192 31296 raft_consensus.cc:374] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:14.505232 31296 raft_consensus.cc:729] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Initialized, Role: FOLLOWER
I20240502 07:59:14.505369 31296 consensus_queue.cc:260] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.505468 31296 raft_consensus.cc:388] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:14.505514 31296 raft_consensus.cc:482] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:14.505571 31296 raft_consensus.cc:3037] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:14.506495 31296 raft_consensus.cc:504] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.506629 31296 leader_election.cc:302] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 050007925f4f41788b9a6bc228db41b0; no voters:
I20240502 07:59:14.506760 31296 leader_election.cc:288] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:14.506822 31299 raft_consensus.cc:2781] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:14.506924 31299 raft_consensus.cc:686] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Becoming Leader. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Running, Role: LEADER
I20240502 07:59:14.507054 31299 consensus_queue.cc:237] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.507302 31296 ts_tablet_manager.cc:1398] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 07:59:14.507825 31296 tablet_bootstrap.cc:492] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0: Bootstrap starting.
I20240502 07:59:14.508456 30684 catalog_manager.cc:5484] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 050007925f4f41788b9a6bc228db41b0 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:14.508832 31296 tablet_bootstrap.cc:654] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:14.509953 31296 tablet_bootstrap.cc:492] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0: No bootstrap required, opened a new log
I20240502 07:59:14.510039 31296 ts_tablet_manager.cc:1367] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:14.510527 31296 raft_consensus.cc:348] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.510629 31296 raft_consensus.cc:374] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:14.510664 31296 raft_consensus.cc:729] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Initialized, Role: FOLLOWER
I20240502 07:59:14.510828 31296 consensus_queue.cc:260] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.510901 31296 raft_consensus.cc:388] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:14.510931 31296 raft_consensus.cc:482] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:14.510968 31296 raft_consensus.cc:3037] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:14.512215 31296 raft_consensus.cc:504] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.512357 31296 leader_election.cc:302] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 050007925f4f41788b9a6bc228db41b0; no voters:
I20240502 07:59:14.512472 31296 leader_election.cc:288] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:14.512518 31299 raft_consensus.cc:2781] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:14.512619 31299 raft_consensus.cc:686] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Becoming Leader. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Running, Role: LEADER
I20240502 07:59:14.512688 31296 ts_tablet_manager.cc:1398] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 07:59:14.512732 31299 consensus_queue.cc:237] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.512900 31296 tablet_bootstrap.cc:492] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0: Bootstrap starting.
I20240502 07:59:14.513727 31296 tablet_bootstrap.cc:654] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:14.514778 31296 tablet_bootstrap.cc:492] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0: No bootstrap required, opened a new log
I20240502 07:59:14.514866 31296 ts_tablet_manager.cc:1367] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:14.515218 30684 catalog_manager.cc:5484] T 6fdbdb698ee64caf9e478368aba84e60 P 050007925f4f41788b9a6bc228db41b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 050007925f4f41788b9a6bc228db41b0 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:14.515309 31296 raft_consensus.cc:348] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.515378 31296 raft_consensus.cc:374] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:14.515414 31296 raft_consensus.cc:729] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Initialized, Role: FOLLOWER
I20240502 07:59:14.515525 31296 consensus_queue.cc:260] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.515605 31296 raft_consensus.cc:388] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:14.515648 31296 raft_consensus.cc:482] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:14.515700 31296 raft_consensus.cc:3037] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:14.516532 31296 raft_consensus.cc:504] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.516659 31296 leader_election.cc:302] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 050007925f4f41788b9a6bc228db41b0; no voters:
I20240502 07:59:14.516794 31298 raft_consensus.cc:2781] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:14.516901 31298 raft_consensus.cc:686] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Becoming Leader. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Running, Role: LEADER
I20240502 07:59:14.517028 31298 consensus_queue.cc:237] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } }
I20240502 07:59:14.517386 31296 leader_election.cc:288] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:14.517640 31296 ts_tablet_manager.cc:1398] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:59:14.518409 30684 catalog_manager.cc:5484] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 050007925f4f41788b9a6bc228db41b0 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:14.680147 30685 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49212:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:14.685747 30936 tablet_service.cc:1467] Processing CreateTablet for tablet 53ae7b8fde174b12948b0d439d38851f (DEFAULT_TABLE table=rebalance_test_table_3 [id=e04c56fdd22b4f57bcb56d168d3ed16a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:14.685782 30937 tablet_service.cc:1467] Processing CreateTablet for tablet 00800e0317b8402a864b8df5ce3c425a (DEFAULT_TABLE table=rebalance_test_table_3 [id=e04c56fdd22b4f57bcb56d168d3ed16a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:14.685951 30938 tablet_service.cc:1467] Processing CreateTablet for tablet 01800acde3ab4f5e849089ad6a0cb645 (DEFAULT_TABLE table=rebalance_test_table_3 [id=e04c56fdd22b4f57bcb56d168d3ed16a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:14.686074 30936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53ae7b8fde174b12948b0d439d38851f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:14.686188 30937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00800e0317b8402a864b8df5ce3c425a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:14.686338 30938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01800acde3ab4f5e849089ad6a0cb645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:14.688450 31326 tablet_bootstrap.cc:492] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833: Bootstrap starting.
I20240502 07:59:14.689280 31326 tablet_bootstrap.cc:654] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:14.690245 31326 tablet_bootstrap.cc:492] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833: No bootstrap required, opened a new log
I20240502 07:59:14.690326 31326 ts_tablet_manager.cc:1367] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:14.690714 31326 raft_consensus.cc:348] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.690815 31326 raft_consensus.cc:374] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:14.690850 31326 raft_consensus.cc:729] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Initialized, Role: FOLLOWER
I20240502 07:59:14.690980 31326 consensus_queue.cc:260] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.691067 31326 raft_consensus.cc:388] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:14.691113 31326 raft_consensus.cc:482] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:14.691157 31326 raft_consensus.cc:3037] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:14.691975 31326 raft_consensus.cc:504] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.692090 31326 leader_election.cc:302] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f1fe3e3d9d34dd88971628d6018d833; no voters:
I20240502 07:59:14.692240 31326 leader_election.cc:288] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:14.692302 31328 raft_consensus.cc:2781] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:14.692399 31328 raft_consensus.cc:686] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Becoming Leader. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Running, Role: LEADER
I20240502 07:59:14.692507 31328 consensus_queue.cc:237] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.692796 31326 ts_tablet_manager.cc:1398] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 07:59:14.692940 31326 tablet_bootstrap.cc:492] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833: Bootstrap starting.
I20240502 07:59:14.694150 31326 tablet_bootstrap.cc:654] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:14.694442 30685 catalog_manager.cc:5484] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "9f1fe3e3d9d34dd88971628d6018d833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:14.700378 31326 tablet_bootstrap.cc:492] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833: No bootstrap required, opened a new log
I20240502 07:59:14.700471 31326 ts_tablet_manager.cc:1367] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240502 07:59:14.700866 31326 raft_consensus.cc:348] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.700953 31326 raft_consensus.cc:374] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:14.700987 31326 raft_consensus.cc:729] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Initialized, Role: FOLLOWER
I20240502 07:59:14.701143 31326 consensus_queue.cc:260] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.701221 31326 raft_consensus.cc:388] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:14.701263 31326 raft_consensus.cc:482] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:14.701330 31326 raft_consensus.cc:3037] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:14.702188 31326 raft_consensus.cc:504] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.702311 31326 leader_election.cc:302] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f1fe3e3d9d34dd88971628d6018d833; no voters:
I20240502 07:59:14.702401 31326 leader_election.cc:288] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:14.702473 31328 raft_consensus.cc:2781] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:14.702582 31328 raft_consensus.cc:686] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Becoming Leader. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Running, Role: LEADER
I20240502 07:59:14.702697 31328 consensus_queue.cc:237] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.702863 31326 ts_tablet_manager.cc:1398] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 07:59:14.703011 31326 tablet_bootstrap.cc:492] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833: Bootstrap starting.
I20240502 07:59:14.704002 31326 tablet_bootstrap.cc:654] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:14.704005 30685 catalog_manager.cc:5484] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "9f1fe3e3d9d34dd88971628d6018d833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:14.704967 31326 tablet_bootstrap.cc:492] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833: No bootstrap required, opened a new log
I20240502 07:59:14.705042 31326 ts_tablet_manager.cc:1367] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:14.705453 31326 raft_consensus.cc:348] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.705538 31326 raft_consensus.cc:374] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:14.705569 31326 raft_consensus.cc:729] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Initialized, Role: FOLLOWER
I20240502 07:59:14.705678 31326 consensus_queue.cc:260] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.705744 31326 raft_consensus.cc:388] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:14.705781 31326 raft_consensus.cc:482] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:14.705827 31326 raft_consensus.cc:3037] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:14.706705 31326 raft_consensus.cc:504] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.706815 31326 leader_election.cc:302] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f1fe3e3d9d34dd88971628d6018d833; no voters:
I20240502 07:59:14.706902 31326 leader_election.cc:288] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:14.707015 31329 raft_consensus.cc:2781] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:14.707091 31329 raft_consensus.cc:686] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Becoming Leader. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Running, Role: LEADER
I20240502 07:59:14.707125 31326 ts_tablet_manager.cc:1398] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:14.707198 31329 consensus_queue.cc:237] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } }
I20240502 07:59:14.708405 30685 catalog_manager.cc:5484] T 00800e0317b8402a864b8df5ce3c425a P 9f1fe3e3d9d34dd88971628d6018d833 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130). New cstate: current_term: 1 leader_uuid: "9f1fe3e3d9d34dd88971628d6018d833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:14.867594 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.131:38213
--local_ip_for_outbound_sockets=127.18.13.131
--tserver_master_addrs=127.18.13.190:34323
--webserver_port=45687
--webserver_interface=127.18.13.131
--builtin_ntp_servers=127.18.13.148:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 07:59:14.975024 31402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:14.975260 31402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:14.975303 31402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:14.978407 31402 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:14.978473 31402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:14.978538 31402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.131
I20240502 07:59:14.982076 31402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.131:38213
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.18.13.131
--webserver_port=45687
--tserver_master_addrs=127.18.13.190:34323
--never_fsync=true
--heap_profile_path=/tmp/kudu.31402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.131
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:14.982975 31402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:14.984172 31402 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:14.992683 31408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:14.992786 31410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:14.993239 31407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:14.994019 31402 server_base.cc:1034] running on GCE node
I20240502 07:59:14.994379 31402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:14.994944 31402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:14.996088 31402 hybrid_clock.cc:648] HybridClock initialized: now 1714636754996071 us; error 29 us; skew 500 ppm
I20240502 07:59:14.997869 31402 webserver.cc:460] Webserver started at http://127.18.13.131:45687/ using document root <none> and password file <none>
I20240502 07:59:14.998411 31402 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:14.998484 31402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:15.001822 31402 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:15.003743 31416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:15.004856 31402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:15.005048 31402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c6642c4ab0104a9da2b204f9e965abfd"
format_stamp: "Formatted at 2024-05-02 07:59:10 on dist-test-slave-f46z"
I20240502 07:59:15.005491 31402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:15.021220 31402 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:15.021806 31402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:15.021960 31402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:15.022469 31402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:15.023356 31402 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:15.023407 31402 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:15.023466 31402 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:15.023494 31402 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:15.032070 31402 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.131:38213
I20240502 07:59:15.032091 31529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.131:38213 every 8 connection(s)
I20240502 07:59:15.033113 31402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 07:59:15.041141 31530 heartbeater.cc:347] Connected to a master server at 127.18.13.190:34323
I20240502 07:59:15.041353 31530 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:15.041834 31530 heartbeater.cc:511] Master 127.18.13.190:34323 requested a full tablet report, sending...
I20240502 07:59:15.042815 30685 ts_manager.cc:194] Re-registered known tserver with Master: c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131:38213)
I20240502 07:59:15.043280 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 31402
I20240502 07:59:15.043377 30685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.131:38041
I20240502 07:59:15.043768 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.132:38863
--local_ip_for_outbound_sockets=127.18.13.132
--tserver_master_addrs=127.18.13.190:34323
--webserver_port=46147
--webserver_interface=127.18.13.132
--builtin_ntp_servers=127.18.13.148:36333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 07:59:15.156706 31533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:15.156955 31533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:15.157018 31533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:15.160188 31533 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:15.160250 31533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:15.160359 31533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.132
I20240502 07:59:15.164216 31533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:36333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.132:38863
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.18.13.132
--webserver_port=46147
--tserver_master_addrs=127.18.13.190:34323
--never_fsync=true
--heap_profile_path=/tmp/kudu.31533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.132
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:15.165305 31533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:15.166466 31533 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:15.174881 31539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:15.175578 31538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:15.176294 31541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:15.176976 31533 server_base.cc:1034] running on GCE node
I20240502 07:59:15.177338 31533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:15.178014 31533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:15.179164 31533 hybrid_clock.cc:648] HybridClock initialized: now 1714636755179149 us; error 32 us; skew 500 ppm
I20240502 07:59:15.180884 31533 webserver.cc:460] Webserver started at http://127.18.13.132:46147/ using document root <none> and password file <none>
I20240502 07:59:15.181409 31533 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:15.181488 31533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:15.184723 31533 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 07:59:15.186537 31547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:15.187469 31533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 07:59:15.187572 31533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "9b205acd80ed40cdad98e381fb296499"
format_stamp: "Formatted at 2024-05-02 07:59:10 on dist-test-slave-f46z"
I20240502 07:59:15.188021 31533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:15.211752 31533 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:15.212424 31533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:15.212582 31533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:15.213102 31533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:15.213994 31533 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:15.214042 31533 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:15.214103 31533 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:15.214128 31533 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:15.222417 31533 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.132:38863
I20240502 07:59:15.222455 31660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.132:38863 every 8 connection(s)
I20240502 07:59:15.223433 31533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:59:15.229559 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 31533
I20240502 07:59:15.232594 31661 heartbeater.cc:347] Connected to a master server at 127.18.13.190:34323
I20240502 07:59:15.232834 31661 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:15.233466 31661 heartbeater.cc:511] Master 127.18.13.190:34323 requested a full tablet report, sending...
I20240502 07:59:15.234593 30685 ts_manager.cc:194] Re-registered known tserver with Master: 9b205acd80ed40cdad98e381fb296499 (127.18.13.132:38863)
I20240502 07:59:15.235420 30685 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.132:32897
I20240502 07:59:15.417024 30938 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:59:15.417814 31595 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:15.418303 30807 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:15.419339 31464 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:15.515064 30807 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:15.516342 31595 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:15.520355 31464 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:15.522995 30938 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:59:15.631863 30807 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:15.633410 31464 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:15.633556 31595 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:15.634191 30938 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 07:59:15.670423 30827 consensus_queue.cc:237] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: true } }
I20240502 07:59:15.671917 31778 raft_consensus.cc:2932] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 9b205acd80ed40cdad98e381fb296499 (127.18.13.132) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: true } } }
I20240502 07:59:15.674968 30681 catalog_manager.cc:5484] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 reported cstate change: config changed from index -1 to 3, NON_VOTER 9b205acd80ed40cdad98e381fb296499 (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:15.676679 30760 consensus_peers.cc:485] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 -> Peer 9b205acd80ed40cdad98e381fb296499 (127.18.13.132:38863): Couldn't send request to peer 9b205acd80ed40cdad98e381fb296499. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b9fbec1717044a5839d0d0f2c967f6f. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:15.682484 30958 consensus_queue.cc:237] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: NON_VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: true } }
I20240502 07:59:15.683988 31789 raft_consensus.cc:2932] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: NON_VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: true } } }
I20240502 07:59:15.688642 30684 catalog_manager.cc:5484] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 reported cstate change: config changed from index -1 to 2, NON_VOTER c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131) added. New cstate: current_term: 1 leader_uuid: "9f1fe3e3d9d34dd88971628d6018d833" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: NON_VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:15.690004 30892 consensus_peers.cc:485] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 -> Peer c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131:38213): Couldn't send request to peer c6642c4ab0104a9da2b204f9e965abfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01800acde3ab4f5e849089ad6a0cb645. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:15.696274 30827 consensus_queue.cc:237] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: true } }
I20240502 07:59:15.697213 31778 raft_consensus.cc:2932] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 9b205acd80ed40cdad98e381fb296499 (127.18.13.132) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: true } } }
W20240502 07:59:15.697589 30760 consensus_peers.cc:485] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 -> Peer 9b205acd80ed40cdad98e381fb296499 (127.18.13.132:38863): Couldn't send request to peer 9b205acd80ed40cdad98e381fb296499. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3741c7c8018b412da5fd73fbbf052102. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:15.699398 30684 catalog_manager.cc:5484] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 reported cstate change: config changed from index -1 to 3, NON_VOTER 9b205acd80ed40cdad98e381fb296499 (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:15.705951 30958 consensus_queue.cc:237] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: true } }
I20240502 07:59:15.706910 31790 raft_consensus.cc:2932] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: true } } }
I20240502 07:59:15.709337 30684 catalog_manager.cc:5484] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 reported cstate change: config changed from index -1 to 3, NON_VOTER 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) added. New cstate: current_term: 1 leader_uuid: "9f1fe3e3d9d34dd88971628d6018d833" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:15.711371 30891 consensus_peers.cc:485] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 -> Peer 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133): Couldn't send request to peer 050007925f4f41788b9a6bc228db41b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d43114a3f804c84b451505f967abe5c. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:15.716720 30827 consensus_queue.cc:237] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: NON_VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: true } }
I20240502 07:59:15.717634 31798 raft_consensus.cc:2932] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: NON_VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: true } } }
I20240502 07:59:15.720755 30684 catalog_manager.cc:5484] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 reported cstate change: config changed from index -1 to 2, NON_VOTER c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131) added. New cstate: current_term: 1 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: NON_VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:15.721463 30761 consensus_peers.cc:485] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 -> Peer c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131:38213): Couldn't send request to peer c6642c4ab0104a9da2b204f9e965abfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39209b4f0a1444b6b80a155657217fe8. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:15.728964 30958 consensus_queue.cc:237] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: true } }
I20240502 07:59:15.729914 31789 raft_consensus.cc:2932] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9b205acd80ed40cdad98e381fb296499 (127.18.13.132) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: true } } }
I20240502 07:59:15.732426 30684 catalog_manager.cc:5484] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 reported cstate change: config changed from index -1 to 2, NON_VOTER 9b205acd80ed40cdad98e381fb296499 (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "9f1fe3e3d9d34dd88971628d6018d833" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:15.733975 30891 consensus_peers.cc:485] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 -> Peer 9b205acd80ed40cdad98e381fb296499 (127.18.13.132:38863): Couldn't send request to peer 9b205acd80ed40cdad98e381fb296499. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa53770eb24c49938a6b0f56bdb85d24. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:15.740474 30827 consensus_queue.cc:237] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: NON_VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: true } }
I20240502 07:59:15.741317 31798 raft_consensus.cc:2932] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: NON_VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: true } } }
W20240502 07:59:15.741786 30761 consensus_peers.cc:485] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 -> Peer c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131:38213): Couldn't send request to peer c6642c4ab0104a9da2b204f9e965abfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de57879c802a4ac08fa6d96efed7360a. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:15.743469 30681 catalog_manager.cc:5484] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 reported cstate change: config changed from index -1 to 2, NON_VOTER c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131) added. New cstate: current_term: 1 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: NON_VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:16.045202 31530 heartbeater.cc:503] Master 127.18.13.190:34323 was elected leader, sending a full tablet report...
I20240502 07:59:16.105086 31893 ts_tablet_manager.cc:902] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd: Initiating tablet copy from peer 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133)
I20240502 07:59:16.105815 31893 tablet_copy_client.cc:308] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:40133
I20240502 07:59:16.109580 30847 tablet_copy_service.cc:140] P 050007925f4f41788b9a6bc228db41b0: Received BeginTabletCopySession request for tablet 39209b4f0a1444b6b80a155657217fe8 from peer c6642c4ab0104a9da2b204f9e965abfd ({username='slave'} at 127.18.13.131:54971)
I20240502 07:59:16.109724 30847 tablet_copy_service.cc:161] P 050007925f4f41788b9a6bc228db41b0: Beginning new tablet copy session on tablet 39209b4f0a1444b6b80a155657217fe8 from peer c6642c4ab0104a9da2b204f9e965abfd at {username='slave'} at 127.18.13.131:54971: session id = c6642c4ab0104a9da2b204f9e965abfd-39209b4f0a1444b6b80a155657217fe8
I20240502 07:59:16.110973 30847 tablet_copy_source_session.cc:205] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:16.113301 31899 ts_tablet_manager.cc:902] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499: Initiating tablet copy from peer 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133)
I20240502 07:59:16.114025 31899 tablet_copy_client.cc:308] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:40133
I20240502 07:59:16.114501 31893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39209b4f0a1444b6b80a155657217fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:16.118618 30847 tablet_copy_service.cc:140] P 050007925f4f41788b9a6bc228db41b0: Received BeginTabletCopySession request for tablet 7b9fbec1717044a5839d0d0f2c967f6f from peer 9b205acd80ed40cdad98e381fb296499 ({username='slave'} at 127.18.13.132:45803)
I20240502 07:59:16.118726 30847 tablet_copy_service.cc:161] P 050007925f4f41788b9a6bc228db41b0: Beginning new tablet copy session on tablet 7b9fbec1717044a5839d0d0f2c967f6f from peer 9b205acd80ed40cdad98e381fb296499 at {username='slave'} at 127.18.13.132:45803: session id = 9b205acd80ed40cdad98e381fb296499-7b9fbec1717044a5839d0d0f2c967f6f
I20240502 07:59:16.119406 30847 tablet_copy_source_session.cc:205] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:16.121320 31893 tablet_copy_client.cc:783] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:16.121498 31893 tablet_copy_client.cc:647] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:16.122298 31899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9fbec1717044a5839d0d0f2c967f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:16.123296 31893 tablet_copy_client.cc:523] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:16.124739 31893 tablet_bootstrap.cc:492] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd: Bootstrap starting.
I20240502 07:59:16.127410 31899 tablet_copy_client.cc:783] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:16.127601 31899 tablet_copy_client.cc:647] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:16.129217 31899 tablet_copy_client.cc:523] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:16.129217 31893 log.cc:824] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:16.130362 31906 ts_tablet_manager.cc:902] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd: Initiating tablet copy from peer 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130:40385)
I20240502 07:59:16.131021 31906 tablet_copy_client.cc:308] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:40385
I20240502 07:59:16.132225 31893 tablet_bootstrap.cc:492] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:16.132532 31893 tablet_bootstrap.cc:492] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd: Bootstrap complete.
I20240502 07:59:16.132826 31893 ts_tablet_manager.cc:1367] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s
I20240502 07:59:16.135355 31899 tablet_bootstrap.cc:492] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499: Bootstrap starting.
I20240502 07:59:16.136397 31893 raft_consensus.cc:348] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: NON_VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: true } }
I20240502 07:59:16.136642 31893 raft_consensus.cc:729] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6642c4ab0104a9da2b204f9e965abfd, State: Initialized, Role: LEARNER
I20240502 07:59:16.137024 31893 consensus_queue.cc:260] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: NON_VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: true } }
I20240502 07:59:16.138938 31893 ts_tablet_manager.cc:1398] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 07:59:16.139640 30847 tablet_copy_service.cc:342] P 050007925f4f41788b9a6bc228db41b0: Request end of tablet copy session c6642c4ab0104a9da2b204f9e965abfd-39209b4f0a1444b6b80a155657217fe8 received from {username='slave'} at 127.18.13.131:54971
I20240502 07:59:16.139752 30847 tablet_copy_service.cc:433] P 050007925f4f41788b9a6bc228db41b0: ending tablet copy session c6642c4ab0104a9da2b204f9e965abfd-39209b4f0a1444b6b80a155657217fe8 on tablet 39209b4f0a1444b6b80a155657217fe8 with peer c6642c4ab0104a9da2b204f9e965abfd
I20240502 07:59:16.141227 30978 tablet_copy_service.cc:140] P 9f1fe3e3d9d34dd88971628d6018d833: Received BeginTabletCopySession request for tablet 01800acde3ab4f5e849089ad6a0cb645 from peer c6642c4ab0104a9da2b204f9e965abfd ({username='slave'} at 127.18.13.131:36327)
I20240502 07:59:16.141338 31899 log.cc:824] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:16.141404 30978 tablet_copy_service.cc:161] P 9f1fe3e3d9d34dd88971628d6018d833: Beginning new tablet copy session on tablet 01800acde3ab4f5e849089ad6a0cb645 from peer c6642c4ab0104a9da2b204f9e965abfd at {username='slave'} at 127.18.13.131:36327: session id = c6642c4ab0104a9da2b204f9e965abfd-01800acde3ab4f5e849089ad6a0cb645
I20240502 07:59:16.142737 30978 tablet_copy_source_session.cc:205] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:16.143882 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01800acde3ab4f5e849089ad6a0cb645. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:16.145643 31906 tablet_copy_client.cc:783] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:16.145805 31906 tablet_copy_client.cc:647] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:16.147168 31906 tablet_copy_client.cc:523] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:16.147960 31899 tablet_bootstrap.cc:492] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:16.148232 31899 tablet_bootstrap.cc:492] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499: Bootstrap complete.
I20240502 07:59:16.148335 31906 tablet_bootstrap.cc:492] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd: Bootstrap starting.
I20240502 07:59:16.148492 31899 ts_tablet_manager.cc:1367] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499: Time spent bootstrapping tablet: real 0.013s user 0.013s sys 0.000s
I20240502 07:59:16.150978 31899 raft_consensus.cc:348] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: true } }
I20240502 07:59:16.151105 31906 tablet_bootstrap.cc:492] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:16.151172 31899 raft_consensus.cc:729] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b205acd80ed40cdad98e381fb296499, State: Initialized, Role: LEARNER
I20240502 07:59:16.151319 31906 tablet_bootstrap.cc:492] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd: Bootstrap complete.
I20240502 07:59:16.151461 31906 ts_tablet_manager.cc:1367] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 07:59:16.151494 31899 consensus_queue.cc:260] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: true } }
I20240502 07:59:16.151983 31906 raft_consensus.cc:348] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: NON_VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: true } }
I20240502 07:59:16.152139 31906 raft_consensus.cc:729] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6642c4ab0104a9da2b204f9e965abfd, State: Initialized, Role: LEARNER
I20240502 07:59:16.152182 31661 heartbeater.cc:503] Master 127.18.13.190:34323 was elected leader, sending a full tablet report...
I20240502 07:59:16.152251 31906 consensus_queue.cc:260] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: NON_VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: true } }
I20240502 07:59:16.152371 31899 ts_tablet_manager.cc:1398] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 07:59:16.152498 31906 ts_tablet_manager.cc:1398] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:16.153301 30847 tablet_copy_service.cc:342] P 050007925f4f41788b9a6bc228db41b0: Request end of tablet copy session 9b205acd80ed40cdad98e381fb296499-7b9fbec1717044a5839d0d0f2c967f6f received from {username='slave'} at 127.18.13.132:45803
I20240502 07:59:16.153400 30847 tablet_copy_service.cc:433] P 050007925f4f41788b9a6bc228db41b0: ending tablet copy session 9b205acd80ed40cdad98e381fb296499-7b9fbec1717044a5839d0d0f2c967f6f on tablet 7b9fbec1717044a5839d0d0f2c967f6f with peer 9b205acd80ed40cdad98e381fb296499
I20240502 07:59:16.154024 30978 tablet_copy_service.cc:342] P 9f1fe3e3d9d34dd88971628d6018d833: Request end of tablet copy session c6642c4ab0104a9da2b204f9e965abfd-01800acde3ab4f5e849089ad6a0cb645 received from {username='slave'} at 127.18.13.131:36327
I20240502 07:59:16.154114 30978 tablet_copy_service.cc:433] P 9f1fe3e3d9d34dd88971628d6018d833: ending tablet copy session c6642c4ab0104a9da2b204f9e965abfd-01800acde3ab4f5e849089ad6a0cb645 on tablet 01800acde3ab4f5e849089ad6a0cb645 with peer c6642c4ab0104a9da2b204f9e965abfd
I20240502 07:59:16.220036 31899 ts_tablet_manager.cc:902] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499: Initiating tablet copy from peer 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130:40385)
I20240502 07:59:16.220535 31899 tablet_copy_client.cc:308] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:40385
I20240502 07:59:16.223986 30978 tablet_copy_service.cc:140] P 9f1fe3e3d9d34dd88971628d6018d833: Received BeginTabletCopySession request for tablet aa53770eb24c49938a6b0f56bdb85d24 from peer 9b205acd80ed40cdad98e381fb296499 ({username='slave'} at 127.18.13.132:41935)
I20240502 07:59:16.224133 30978 tablet_copy_service.cc:161] P 9f1fe3e3d9d34dd88971628d6018d833: Beginning new tablet copy session on tablet aa53770eb24c49938a6b0f56bdb85d24 from peer 9b205acd80ed40cdad98e381fb296499 at {username='slave'} at 127.18.13.132:41935: session id = 9b205acd80ed40cdad98e381fb296499-aa53770eb24c49938a6b0f56bdb85d24
I20240502 07:59:16.224984 30978 tablet_copy_source_session.cc:205] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:16.225997 31899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa53770eb24c49938a6b0f56bdb85d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:16.227965 31899 tablet_copy_client.cc:783] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:16.228133 31899 tablet_copy_client.cc:647] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:16.229123 31899 tablet_copy_client.cc:523] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:16.230463 31899 tablet_bootstrap.cc:492] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499: Bootstrap starting.
I20240502 07:59:16.233887 31899 tablet_bootstrap.cc:492] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:16.234134 31899 tablet_bootstrap.cc:492] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499: Bootstrap complete.
I20240502 07:59:16.234284 31899 ts_tablet_manager.cc:1367] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 07:59:16.234757 31899 raft_consensus.cc:348] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: true } }
I20240502 07:59:16.234885 31899 raft_consensus.cc:729] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b205acd80ed40cdad98e381fb296499, State: Initialized, Role: LEARNER
I20240502 07:59:16.235002 31899 consensus_queue.cc:260] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: true } }
I20240502 07:59:16.235281 31899 ts_tablet_manager.cc:1398] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:16.235886 30978 tablet_copy_service.cc:342] P 9f1fe3e3d9d34dd88971628d6018d833: Request end of tablet copy session 9b205acd80ed40cdad98e381fb296499-aa53770eb24c49938a6b0f56bdb85d24 received from {username='slave'} at 127.18.13.132:41935
I20240502 07:59:16.235951 30978 tablet_copy_service.cc:433] P 9f1fe3e3d9d34dd88971628d6018d833: ending tablet copy session 9b205acd80ed40cdad98e381fb296499-aa53770eb24c49938a6b0f56bdb85d24 on tablet aa53770eb24c49938a6b0f56bdb85d24 with peer 9b205acd80ed40cdad98e381fb296499
I20240502 07:59:16.261173 31899 ts_tablet_manager.cc:902] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499: Initiating tablet copy from peer 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133)
I20240502 07:59:16.261619 31899 tablet_copy_client.cc:308] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:40133
I20240502 07:59:16.262120 30847 tablet_copy_service.cc:140] P 050007925f4f41788b9a6bc228db41b0: Received BeginTabletCopySession request for tablet 3741c7c8018b412da5fd73fbbf052102 from peer 9b205acd80ed40cdad98e381fb296499 ({username='slave'} at 127.18.13.132:45803)
I20240502 07:59:16.262233 30847 tablet_copy_service.cc:161] P 050007925f4f41788b9a6bc228db41b0: Beginning new tablet copy session on tablet 3741c7c8018b412da5fd73fbbf052102 from peer 9b205acd80ed40cdad98e381fb296499 at {username='slave'} at 127.18.13.132:45803: session id = 9b205acd80ed40cdad98e381fb296499-3741c7c8018b412da5fd73fbbf052102
I20240502 07:59:16.263227 30847 tablet_copy_source_session.cc:205] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:16.264104 31899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3741c7c8018b412da5fd73fbbf052102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:16.266258 31899 tablet_copy_client.cc:783] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:16.266419 31899 tablet_copy_client.cc:647] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:16.267468 31899 tablet_copy_client.cc:523] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:16.268715 31899 tablet_bootstrap.cc:492] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499: Bootstrap starting.
I20240502 07:59:16.272670 31899 tablet_bootstrap.cc:492] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:16.272882 31899 tablet_bootstrap.cc:492] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499: Bootstrap complete.
I20240502 07:59:16.273006 31899 ts_tablet_manager.cc:1367] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 07:59:16.273362 31899 raft_consensus.cc:348] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: true } }
I20240502 07:59:16.273466 31899 raft_consensus.cc:729] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b205acd80ed40cdad98e381fb296499, State: Initialized, Role: LEARNER
I20240502 07:59:16.273551 31899 consensus_queue.cc:260] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: true } }
I20240502 07:59:16.273789 31899 ts_tablet_manager.cc:1398] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:16.274399 30847 tablet_copy_service.cc:342] P 050007925f4f41788b9a6bc228db41b0: Request end of tablet copy session 9b205acd80ed40cdad98e381fb296499-3741c7c8018b412da5fd73fbbf052102 received from {username='slave'} at 127.18.13.132:45803
I20240502 07:59:16.274492 30847 tablet_copy_service.cc:433] P 050007925f4f41788b9a6bc228db41b0: ending tablet copy session 9b205acd80ed40cdad98e381fb296499-3741c7c8018b412da5fd73fbbf052102 on tablet 3741c7c8018b412da5fd73fbbf052102 with peer 9b205acd80ed40cdad98e381fb296499
I20240502 07:59:16.309083 31960 ts_tablet_manager.cc:902] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0: Initiating tablet copy from peer 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130:40385)
I20240502 07:59:16.309934 31960 tablet_copy_client.cc:308] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:40385
I20240502 07:59:16.313387 30978 tablet_copy_service.cc:140] P 9f1fe3e3d9d34dd88971628d6018d833: Received BeginTabletCopySession request for tablet 4d43114a3f804c84b451505f967abe5c from peer 050007925f4f41788b9a6bc228db41b0 ({username='slave'} at 127.18.13.129:47267)
I20240502 07:59:16.313494 30978 tablet_copy_service.cc:161] P 9f1fe3e3d9d34dd88971628d6018d833: Beginning new tablet copy session on tablet 4d43114a3f804c84b451505f967abe5c from peer 050007925f4f41788b9a6bc228db41b0 at {username='slave'} at 127.18.13.129:47267: session id = 050007925f4f41788b9a6bc228db41b0-4d43114a3f804c84b451505f967abe5c
I20240502 07:59:16.314200 30978 tablet_copy_source_session.cc:205] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:16.315268 31960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d43114a3f804c84b451505f967abe5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:16.316550 31906 ts_tablet_manager.cc:902] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd: Initiating tablet copy from peer 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133)
I20240502 07:59:16.317055 31906 tablet_copy_client.cc:308] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:40133
I20240502 07:59:16.317550 30847 tablet_copy_service.cc:140] P 050007925f4f41788b9a6bc228db41b0: Received BeginTabletCopySession request for tablet de57879c802a4ac08fa6d96efed7360a from peer c6642c4ab0104a9da2b204f9e965abfd ({username='slave'} at 127.18.13.131:54971)
I20240502 07:59:16.317569 31960 tablet_copy_client.cc:783] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:16.317668 30847 tablet_copy_service.cc:161] P 050007925f4f41788b9a6bc228db41b0: Beginning new tablet copy session on tablet de57879c802a4ac08fa6d96efed7360a from peer c6642c4ab0104a9da2b204f9e965abfd at {username='slave'} at 127.18.13.131:54971: session id = c6642c4ab0104a9da2b204f9e965abfd-de57879c802a4ac08fa6d96efed7360a
I20240502 07:59:16.317804 31960 tablet_copy_client.cc:647] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:16.318528 30847 tablet_copy_source_session.cc:205] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:16.319222 31960 tablet_copy_client.cc:523] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:16.319289 31906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de57879c802a4ac08fa6d96efed7360a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:16.320880 31960 tablet_bootstrap.cc:492] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0: Bootstrap starting.
I20240502 07:59:16.321480 31906 tablet_copy_client.cc:783] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:16.321624 31906 tablet_copy_client.cc:647] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:16.322667 31906 tablet_copy_client.cc:523] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:16.323845 31906 tablet_bootstrap.cc:492] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd: Bootstrap starting.
I20240502 07:59:16.326834 31960 tablet_bootstrap.cc:492] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:16.327160 31960 tablet_bootstrap.cc:492] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0: Bootstrap complete.
I20240502 07:59:16.327273 31906 tablet_bootstrap.cc:492] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:16.327399 31960 ts_tablet_manager.cc:1367] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20240502 07:59:16.327507 31906 tablet_bootstrap.cc:492] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd: Bootstrap complete.
I20240502 07:59:16.327652 31906 ts_tablet_manager.cc:1367] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 07:59:16.327978 31960 raft_consensus.cc:348] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: true } }
I20240502 07:59:16.328099 31960 raft_consensus.cc:729] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Initialized, Role: LEARNER
I20240502 07:59:16.328207 31960 consensus_queue.cc:260] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: true } }
I20240502 07:59:16.328284 31906 raft_consensus.cc:348] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: NON_VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: true } }
I20240502 07:59:16.328498 31960 ts_tablet_manager.cc:1398] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 07:59:16.328382 31906 raft_consensus.cc:729] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6642c4ab0104a9da2b204f9e965abfd, State: Initialized, Role: LEARNER
I20240502 07:59:16.328894 31906 consensus_queue.cc:260] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: NON_VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: true } }
I20240502 07:59:16.329071 30978 tablet_copy_service.cc:342] P 9f1fe3e3d9d34dd88971628d6018d833: Request end of tablet copy session 050007925f4f41788b9a6bc228db41b0-4d43114a3f804c84b451505f967abe5c received from {username='slave'} at 127.18.13.129:47267
I20240502 07:59:16.329154 30978 tablet_copy_service.cc:433] P 9f1fe3e3d9d34dd88971628d6018d833: ending tablet copy session 050007925f4f41788b9a6bc228db41b0-4d43114a3f804c84b451505f967abe5c on tablet 4d43114a3f804c84b451505f967abe5c with peer 050007925f4f41788b9a6bc228db41b0
I20240502 07:59:16.331115 31906 ts_tablet_manager.cc:1398] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 07:59:16.331602 30847 tablet_copy_service.cc:342] P 050007925f4f41788b9a6bc228db41b0: Request end of tablet copy session c6642c4ab0104a9da2b204f9e965abfd-de57879c802a4ac08fa6d96efed7360a received from {username='slave'} at 127.18.13.131:54971
I20240502 07:59:16.331679 30847 tablet_copy_service.cc:433] P 050007925f4f41788b9a6bc228db41b0: ending tablet copy session c6642c4ab0104a9da2b204f9e965abfd-de57879c802a4ac08fa6d96efed7360a on tablet de57879c802a4ac08fa6d96efed7360a with peer c6642c4ab0104a9da2b204f9e965abfd
I20240502 07:59:16.536840 31615 raft_consensus.cc:1203] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240502 07:59:16.607632 31484 raft_consensus.cc:1203] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 07:59:16.687161 31484 raft_consensus.cc:1203] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 07:59:16.710213 31615 raft_consensus.cc:1203] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 07:59:16.777928 31615 raft_consensus.cc:1203] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240502 07:59:16.797237 31484 raft_consensus.cc:1203] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 07:59:16.856209 30827 raft_consensus.cc:1203] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240502 07:59:16.930400 32008 raft_consensus.cc:1050] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0: attempting to promote NON_VOTER 9b205acd80ed40cdad98e381fb296499 to VOTER
I20240502 07:59:16.930878 32008 consensus_queue.cc:237] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } }
I20240502 07:59:16.932102 31615 raft_consensus.cc:1261] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [term 1 LEARNER]: Refusing update from remote peer 050007925f4f41788b9a6bc228db41b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:16.932562 31779 consensus_queue.cc:1035] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:16.935092 32008 raft_consensus.cc:2932] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 9b205acd80ed40cdad98e381fb296499 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } } }
I20240502 07:59:16.935604 31615 raft_consensus.cc:2932] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 9b205acd80ed40cdad98e381fb296499 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } } }
I20240502 07:59:16.937537 30684 catalog_manager.cc:5484] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 reported cstate change: config changed from index 3 to 4, 9b205acd80ed40cdad98e381fb296499 (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:16.980907 30827 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b9fbec1717044a5839d0d0f2c967f6f"
dest_uuid: "050007925f4f41788b9a6bc228db41b0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51872
I20240502 07:59:16.981076 30827 raft_consensus.cc:595] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:17.091611 32008 raft_consensus.cc:1050] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0: attempting to promote NON_VOTER c6642c4ab0104a9da2b204f9e965abfd to VOTER
I20240502 07:59:17.092011 32008 consensus_queue.cc:237] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } }
I20240502 07:59:17.093247 31484 raft_consensus.cc:1261] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [term 1 LEARNER]: Refusing update from remote peer 050007925f4f41788b9a6bc228db41b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 07:59:17.093668 32008 consensus_queue.cc:1035] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 07:59:17.096282 31779 raft_consensus.cc:2932] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } } }
I20240502 07:59:17.096761 31484 raft_consensus.cc:2932] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } } }
I20240502 07:59:17.098676 30681 catalog_manager.cc:5484] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 reported cstate change: config changed from index 2 to 3, c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:17.109902 32010 raft_consensus.cc:1050] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833: attempting to promote NON_VOTER 9b205acd80ed40cdad98e381fb296499 to VOTER
I20240502 07:59:17.110442 32010 consensus_queue.cc:237] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } }
I20240502 07:59:17.111560 31615 raft_consensus.cc:1261] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [term 1 LEARNER]: Refusing update from remote peer 9f1fe3e3d9d34dd88971628d6018d833: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 07:59:17.112025 32009 consensus_queue.cc:1035] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 07:59:17.113729 32010 raft_consensus.cc:2932] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9b205acd80ed40cdad98e381fb296499 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } } }
I20240502 07:59:17.114151 31615 raft_consensus.cc:2932] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9b205acd80ed40cdad98e381fb296499 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } } }
I20240502 07:59:17.116305 30684 catalog_manager.cc:5484] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 reported cstate change: config changed from index 2 to 3, 9b205acd80ed40cdad98e381fb296499 (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9f1fe3e3d9d34dd88971628d6018d833" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } } }
I20240502 07:59:17.231092 30827 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b9fbec1717044a5839d0d0f2c967f6f"
dest_uuid: "050007925f4f41788b9a6bc228db41b0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51920
I20240502 07:59:17.231231 30827 raft_consensus.cc:595] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:17.245854 30958 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa53770eb24c49938a6b0f56bdb85d24"
dest_uuid: "9f1fe3e3d9d34dd88971628d6018d833"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44794
I20240502 07:59:17.246007 30958 raft_consensus.cc:595] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:17.262212 32009 raft_consensus.cc:1050] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833: attempting to promote NON_VOTER 050007925f4f41788b9a6bc228db41b0 to VOTER
I20240502 07:59:17.262598 32009 consensus_queue.cc:237] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } }
I20240502 07:59:17.262629 30827 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "39209b4f0a1444b6b80a155657217fe8"
dest_uuid: "050007925f4f41788b9a6bc228db41b0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51948
I20240502 07:59:17.262748 30827 raft_consensus.cc:595] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:17.263891 30827 raft_consensus.cc:1261] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [term 1 LEARNER]: Refusing update from remote peer 9f1fe3e3d9d34dd88971628d6018d833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:17.264459 32009 consensus_queue.cc:1035] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:17.266219 32009 raft_consensus.cc:2932] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } } }
I20240502 07:59:17.267011 31790 raft_consensus.cc:1050] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833: attempting to promote NON_VOTER c6642c4ab0104a9da2b204f9e965abfd to VOTER
I20240502 07:59:17.266922 30827 raft_consensus.cc:2932] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } } }
I20240502 07:59:17.267329 31790 consensus_queue.cc:237] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } }
I20240502 07:59:17.268568 31484 raft_consensus.cc:1261] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [term 1 LEARNER]: Refusing update from remote peer 9f1fe3e3d9d34dd88971628d6018d833: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 07:59:17.269239 30681 catalog_manager.cc:5484] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 reported cstate change: config changed from index 3 to 4, 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9f1fe3e3d9d34dd88971628d6018d833" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:17.269795 31790 consensus_queue.cc:1035] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 07:59:17.271869 31790 raft_consensus.cc:2932] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } } }
I20240502 07:59:17.272404 31484 raft_consensus.cc:2932] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } } }
I20240502 07:59:17.273682 30684 catalog_manager.cc:5484] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 reported cstate change: config changed from index 2 to 3, c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9f1fe3e3d9d34dd88971628d6018d833" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:17.382539 32007 raft_consensus.cc:1050] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0: attempting to promote NON_VOTER 9b205acd80ed40cdad98e381fb296499 to VOTER
I20240502 07:59:17.382905 32007 consensus_queue.cc:237] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } }
I20240502 07:59:17.383976 31615 raft_consensus.cc:1261] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [term 1 LEARNER]: Refusing update from remote peer 050007925f4f41788b9a6bc228db41b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:17.384356 32007 consensus_queue.cc:1035] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:17.386005 31779 raft_consensus.cc:2932] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 9b205acd80ed40cdad98e381fb296499 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } } }
I20240502 07:59:17.386466 31615 raft_consensus.cc:2932] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 9b205acd80ed40cdad98e381fb296499 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } } }
I20240502 07:59:17.388491 30681 catalog_manager.cc:5484] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 reported cstate change: config changed from index 3 to 4, 9b205acd80ed40cdad98e381fb296499 (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:17.405369 32008 raft_consensus.cc:1050] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0: attempting to promote NON_VOTER c6642c4ab0104a9da2b204f9e965abfd to VOTER
I20240502 07:59:17.405694 32008 consensus_queue.cc:237] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } }
I20240502 07:59:17.406728 31484 raft_consensus.cc:1261] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [term 1 LEARNER]: Refusing update from remote peer 050007925f4f41788b9a6bc228db41b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 07:59:17.407104 32008 consensus_queue.cc:1035] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 07:59:17.408742 32007 raft_consensus.cc:2932] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } } }
I20240502 07:59:17.409189 31484 raft_consensus.cc:2932] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } } }
I20240502 07:59:17.411047 30684 catalog_manager.cc:5484] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 reported cstate change: config changed from index 2 to 3, c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:17.474606 30827 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "de57879c802a4ac08fa6d96efed7360a"
dest_uuid: "050007925f4f41788b9a6bc228db41b0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51976
I20240502 07:59:17.474741 30827 raft_consensus.cc:595] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:17.485271 30958 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "01800acde3ab4f5e849089ad6a0cb645"
dest_uuid: "9f1fe3e3d9d34dd88971628d6018d833"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44816
I20240502 07:59:17.485392 30958 raft_consensus.cc:595] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:17.495533 30827 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b9fbec1717044a5839d0d0f2c967f6f"
dest_uuid: "050007925f4f41788b9a6bc228db41b0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51994
I20240502 07:59:17.495653 30827 raft_consensus.cc:595] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:17.505208 30827 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3741c7c8018b412da5fd73fbbf052102"
dest_uuid: "050007925f4f41788b9a6bc228db41b0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52012
I20240502 07:59:17.505329 30827 raft_consensus.cc:595] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:17.515920 30958 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa53770eb24c49938a6b0f56bdb85d24"
dest_uuid: "9f1fe3e3d9d34dd88971628d6018d833"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44844
I20240502 07:59:17.516054 30958 raft_consensus.cc:595] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:17.525933 30958 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4d43114a3f804c84b451505f967abe5c"
dest_uuid: "9f1fe3e3d9d34dd88971628d6018d833"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44874
I20240502 07:59:17.526062 30958 raft_consensus.cc:595] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:17.531404 31779 raft_consensus.cc:979] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0: : Instructing follower 9b205acd80ed40cdad98e381fb296499 to start an election
I20240502 07:59:17.531515 31779 raft_consensus.cc:1067] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Signalling peer 9b205acd80ed40cdad98e381fb296499 to start an election
I20240502 07:59:17.532136 31615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b9fbec1717044a5839d0d0f2c967f6f"
dest_uuid: "9b205acd80ed40cdad98e381fb296499"
from {username='slave'} at 127.18.13.129:54901
I20240502 07:59:17.532317 31615 raft_consensus.cc:482] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:17.532436 31615 raft_consensus.cc:3037] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:17.533317 31615 raft_consensus.cc:504] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } }
I20240502 07:59:17.534175 31615 leader_election.cc:288] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [CANDIDATE]: Term 2 election: Requested vote from peers 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133)
I20240502 07:59:17.534621 30827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9fbec1717044a5839d0d0f2c967f6f" candidate_uuid: "9b205acd80ed40cdad98e381fb296499" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "050007925f4f41788b9a6bc228db41b0"
I20240502 07:59:17.534804 30827 raft_consensus.cc:3032] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:17.534858 30827 raft_consensus.cc:729] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Running, Role: LEADER
I20240502 07:59:17.535010 30827 consensus_queue.cc:260] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } }
I20240502 07:59:17.535174 30827 raft_consensus.cc:3037] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:17.536051 30827 raft_consensus.cc:2454] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b205acd80ed40cdad98e381fb296499 in term 2.
I20240502 07:59:17.536412 31548 leader_election.cc:302] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 050007925f4f41788b9a6bc228db41b0, 9b205acd80ed40cdad98e381fb296499; no voters:
I20240502 07:59:17.536767 31914 raft_consensus.cc:2781] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:17.536968 31914 raft_consensus.cc:686] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [term 2 LEADER]: Becoming Leader. State: Replica: 9b205acd80ed40cdad98e381fb296499, State: Running, Role: LEADER
I20240502 07:59:17.537262 31914 consensus_queue.cc:237] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } }
I20240502 07:59:17.538473 30827 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "39209b4f0a1444b6b80a155657217fe8"
dest_uuid: "050007925f4f41788b9a6bc228db41b0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52026
I20240502 07:59:17.538594 30827 raft_consensus.cc:595] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:17.539155 30681 catalog_manager.cc:5484] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 reported cstate change: term changed from 1 to 2, leader changed from 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) to 9b205acd80ed40cdad98e381fb296499 (127.18.13.132). New cstate: current_term: 2 leader_uuid: "9b205acd80ed40cdad98e381fb296499" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 07:59:17.545292 30669 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 7b9fbec1717044a5839d0d0f2c967f6f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 9b205acd80ed40cdad98e381fb296499 (127.18.13.132:38863) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 07:59:17.639360 32007 raft_consensus.cc:979] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0: : Instructing follower c6642c4ab0104a9da2b204f9e965abfd to start an election
I20240502 07:59:17.639492 32007 raft_consensus.cc:1067] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Signalling peer c6642c4ab0104a9da2b204f9e965abfd to start an election
I20240502 07:59:17.640033 31484 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "39209b4f0a1444b6b80a155657217fe8"
dest_uuid: "c6642c4ab0104a9da2b204f9e965abfd"
from {username='slave'} at 127.18.13.129:51539
I20240502 07:59:17.640255 31484 raft_consensus.cc:482] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:17.640375 31484 raft_consensus.cc:3037] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:17.641307 31484 raft_consensus.cc:504] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } }
I20240502 07:59:17.642151 31484 leader_election.cc:288] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [CANDIDATE]: Term 2 election: Requested vote from peers 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133)
I20240502 07:59:17.642537 30827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39209b4f0a1444b6b80a155657217fe8" candidate_uuid: "c6642c4ab0104a9da2b204f9e965abfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "050007925f4f41788b9a6bc228db41b0"
I20240502 07:59:17.642669 30827 raft_consensus.cc:3032] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:17.642803 30827 raft_consensus.cc:729] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Running, Role: LEADER
I20240502 07:59:17.642997 30827 consensus_queue.cc:260] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } }
I20240502 07:59:17.643131 30827 raft_consensus.cc:3037] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:17.643934 30827 raft_consensus.cc:2454] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6642c4ab0104a9da2b204f9e965abfd in term 2.
I20240502 07:59:17.644271 31417 leader_election.cc:302] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 050007925f4f41788b9a6bc228db41b0, c6642c4ab0104a9da2b204f9e965abfd; no voters:
I20240502 07:59:17.644503 31911 raft_consensus.cc:2781] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:17.644670 31911 raft_consensus.cc:686] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [term 2 LEADER]: Becoming Leader. State: Replica: c6642c4ab0104a9da2b204f9e965abfd, State: Running, Role: LEADER
I20240502 07:59:17.644946 31911 consensus_queue.cc:237] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } }
I20240502 07:59:17.646725 30681 catalog_manager.cc:5484] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd reported cstate change: term changed from 1 to 2, leader changed from 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) to c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131). New cstate: current_term: 2 leader_uuid: "c6642c4ab0104a9da2b204f9e965abfd" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:17.725958 32148 raft_consensus.cc:979] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833: : Instructing follower c6642c4ab0104a9da2b204f9e965abfd to start an election
I20240502 07:59:17.726092 32148 raft_consensus.cc:1067] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Signalling peer c6642c4ab0104a9da2b204f9e965abfd to start an election
I20240502 07:59:17.726670 31484 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "01800acde3ab4f5e849089ad6a0cb645"
dest_uuid: "c6642c4ab0104a9da2b204f9e965abfd"
from {username='slave'} at 127.18.13.130:37033
I20240502 07:59:17.726814 31484 raft_consensus.cc:482] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:17.726886 31484 raft_consensus.cc:3037] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:17.727744 31484 raft_consensus.cc:504] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } }
I20240502 07:59:17.728134 31484 leader_election.cc:288] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [CANDIDATE]: Term 2 election: Requested vote from peers 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130:40385)
I20240502 07:59:17.728590 30958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01800acde3ab4f5e849089ad6a0cb645" candidate_uuid: "c6642c4ab0104a9da2b204f9e965abfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9f1fe3e3d9d34dd88971628d6018d833"
I20240502 07:59:17.728765 30958 raft_consensus.cc:3032] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:17.728814 30958 raft_consensus.cc:729] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Running, Role: LEADER
I20240502 07:59:17.728958 30958 consensus_queue.cc:260] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } }
I20240502 07:59:17.729103 30958 raft_consensus.cc:3037] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:17.730000 30958 raft_consensus.cc:2454] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6642c4ab0104a9da2b204f9e965abfd in term 2.
I20240502 07:59:17.730336 31418 leader_election.cc:302] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9f1fe3e3d9d34dd88971628d6018d833, c6642c4ab0104a9da2b204f9e965abfd; no voters:
I20240502 07:59:17.730496 31911 raft_consensus.cc:2781] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:17.730602 31911 raft_consensus.cc:686] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [term 2 LEADER]: Becoming Leader. State: Replica: c6642c4ab0104a9da2b204f9e965abfd, State: Running, Role: LEADER
I20240502 07:59:17.730762 31911 consensus_queue.cc:237] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } }
I20240502 07:59:17.732197 30681 catalog_manager.cc:5484] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd reported cstate change: term changed from 1 to 2, leader changed from 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130) to c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131). New cstate: current_term: 2 leader_uuid: "c6642c4ab0104a9da2b204f9e965abfd" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:17.734386 32148 raft_consensus.cc:979] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833: : Instructing follower 9b205acd80ed40cdad98e381fb296499 to start an election
I20240502 07:59:17.734504 32010 raft_consensus.cc:1067] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Signalling peer 9b205acd80ed40cdad98e381fb296499 to start an election
I20240502 07:59:17.734934 31615 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa53770eb24c49938a6b0f56bdb85d24"
dest_uuid: "9b205acd80ed40cdad98e381fb296499"
from {username='slave'} at 127.18.13.130:34099
I20240502 07:59:17.735054 31615 raft_consensus.cc:482] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:17.735119 31615 raft_consensus.cc:3037] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:17.735864 31615 raft_consensus.cc:504] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } }
I20240502 07:59:17.736186 31615 leader_election.cc:288] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [CANDIDATE]: Term 2 election: Requested vote from peers 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130:40385)
I20240502 07:59:17.736541 30958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa53770eb24c49938a6b0f56bdb85d24" candidate_uuid: "9b205acd80ed40cdad98e381fb296499" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "9f1fe3e3d9d34dd88971628d6018d833"
I20240502 07:59:17.736647 30958 raft_consensus.cc:3032] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:17.736698 30958 raft_consensus.cc:729] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Running, Role: LEADER
I20240502 07:59:17.736833 30958 consensus_queue.cc:260] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } }
I20240502 07:59:17.736963 30958 raft_consensus.cc:3037] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:17.737668 30958 raft_consensus.cc:2454] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b205acd80ed40cdad98e381fb296499 in term 2.
I20240502 07:59:17.737941 31549 leader_election.cc:302] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9b205acd80ed40cdad98e381fb296499, 9f1fe3e3d9d34dd88971628d6018d833; no voters:
I20240502 07:59:17.738113 31914 raft_consensus.cc:2781] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:17.738204 31914 raft_consensus.cc:686] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [term 2 LEADER]: Becoming Leader. State: Replica: 9b205acd80ed40cdad98e381fb296499, State: Running, Role: LEADER
I20240502 07:59:17.738334 31914 consensus_queue.cc:237] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } }
I20240502 07:59:17.739531 30681 catalog_manager.cc:5484] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 reported cstate change: term changed from 1 to 2, leader changed from 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130) to 9b205acd80ed40cdad98e381fb296499 (127.18.13.132). New cstate: current_term: 2 leader_uuid: "9b205acd80ed40cdad98e381fb296499" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:17.750279 30827 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "de57879c802a4ac08fa6d96efed7360a"
dest_uuid: "050007925f4f41788b9a6bc228db41b0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52042
I20240502 07:59:17.750413 30827 raft_consensus.cc:595] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:17.770071 32148 raft_consensus.cc:979] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833: : Instructing follower 050007925f4f41788b9a6bc228db41b0 to start an election
I20240502 07:59:17.770186 32010 raft_consensus.cc:1067] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Signalling peer 050007925f4f41788b9a6bc228db41b0 to start an election
I20240502 07:59:17.770648 30827 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d43114a3f804c84b451505f967abe5c"
dest_uuid: "050007925f4f41788b9a6bc228db41b0"
from {username='slave'} at 127.18.13.130:52151
I20240502 07:59:17.770797 30827 raft_consensus.cc:482] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:17.770864 30827 raft_consensus.cc:3037] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:17.771869 30827 raft_consensus.cc:504] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } }
I20240502 07:59:17.772444 30827 leader_election.cc:288] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 2 election: Requested vote from peers 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130:40385)
I20240502 07:59:17.772828 30958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d43114a3f804c84b451505f967abe5c" candidate_uuid: "050007925f4f41788b9a6bc228db41b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9f1fe3e3d9d34dd88971628d6018d833"
I20240502 07:59:17.772969 30958 raft_consensus.cc:3032] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:17.773041 30958 raft_consensus.cc:729] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Running, Role: LEADER
I20240502 07:59:17.773207 30958 consensus_queue.cc:260] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } }
I20240502 07:59:17.773362 30958 raft_consensus.cc:3037] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:17.773911 30827 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3741c7c8018b412da5fd73fbbf052102"
dest_uuid: "050007925f4f41788b9a6bc228db41b0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52062
I20240502 07:59:17.774006 30827 raft_consensus.cc:595] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:17.774327 30958 raft_consensus.cc:2454] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 050007925f4f41788b9a6bc228db41b0 in term 2.
I20240502 07:59:17.774691 30761 leader_election.cc:302] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 050007925f4f41788b9a6bc228db41b0, 9f1fe3e3d9d34dd88971628d6018d833; no voters:
I20240502 07:59:17.774955 32007 raft_consensus.cc:2781] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:17.775053 32007 raft_consensus.cc:686] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [term 2 LEADER]: Becoming Leader. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Running, Role: LEADER
I20240502 07:59:17.775195 32007 consensus_queue.cc:237] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } }
I20240502 07:59:17.776983 30681 catalog_manager.cc:5484] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 reported cstate change: term changed from 1 to 2, leader changed from 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130) to 050007925f4f41788b9a6bc228db41b0 (127.18.13.129). New cstate: current_term: 2 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:17.828565 32007 raft_consensus.cc:979] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0: : Instructing follower c6642c4ab0104a9da2b204f9e965abfd to start an election
I20240502 07:59:17.828706 32007 raft_consensus.cc:1067] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Signalling peer c6642c4ab0104a9da2b204f9e965abfd to start an election
I20240502 07:59:17.829133 31484 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de57879c802a4ac08fa6d96efed7360a"
dest_uuid: "c6642c4ab0104a9da2b204f9e965abfd"
from {username='slave'} at 127.18.13.129:51539
I20240502 07:59:17.829285 31484 raft_consensus.cc:482] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:17.829370 31484 raft_consensus.cc:3037] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:17.830221 31484 raft_consensus.cc:504] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } }
I20240502 07:59:17.830606 31484 leader_election.cc:288] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [CANDIDATE]: Term 2 election: Requested vote from peers 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133)
I20240502 07:59:17.831015 30827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de57879c802a4ac08fa6d96efed7360a" candidate_uuid: "c6642c4ab0104a9da2b204f9e965abfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "050007925f4f41788b9a6bc228db41b0"
I20240502 07:59:17.831122 30827 raft_consensus.cc:3032] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:17.831156 30827 raft_consensus.cc:729] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Running, Role: LEADER
I20240502 07:59:17.831301 30827 consensus_queue.cc:260] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } }
I20240502 07:59:17.831416 30827 raft_consensus.cc:3037] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:17.832170 30827 raft_consensus.cc:2454] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6642c4ab0104a9da2b204f9e965abfd in term 2.
I20240502 07:59:17.832443 31417 leader_election.cc:302] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 050007925f4f41788b9a6bc228db41b0, c6642c4ab0104a9da2b204f9e965abfd; no voters:
I20240502 07:59:17.832659 31911 raft_consensus.cc:2781] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:17.832767 31911 raft_consensus.cc:686] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [term 2 LEADER]: Becoming Leader. State: Replica: c6642c4ab0104a9da2b204f9e965abfd, State: Running, Role: LEADER
I20240502 07:59:17.832897 31911 consensus_queue.cc:237] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } }
I20240502 07:59:17.834247 30681 catalog_manager.cc:5484] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd reported cstate change: term changed from 1 to 2, leader changed from 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) to c6642c4ab0104a9da2b204f9e965abfd (127.18.13.131). New cstate: current_term: 2 leader_uuid: "c6642c4ab0104a9da2b204f9e965abfd" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:18.010504 31779 raft_consensus.cc:979] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0: : Instructing follower 9b205acd80ed40cdad98e381fb296499 to start an election
I20240502 07:59:18.010614 31779 raft_consensus.cc:1067] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Signalling peer 9b205acd80ed40cdad98e381fb296499 to start an election
I20240502 07:59:18.011435 31614 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3741c7c8018b412da5fd73fbbf052102"
dest_uuid: "9b205acd80ed40cdad98e381fb296499"
from {username='slave'} at 127.18.13.129:54901
I20240502 07:59:18.011577 31614 raft_consensus.cc:482] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:18.011644 31614 raft_consensus.cc:3037] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:18.012823 31614 raft_consensus.cc:504] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } }
I20240502 07:59:18.013163 31614 leader_election.cc:288] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [CANDIDATE]: Term 2 election: Requested vote from peers 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133)
I20240502 07:59:18.013602 30827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3741c7c8018b412da5fd73fbbf052102" candidate_uuid: "9b205acd80ed40cdad98e381fb296499" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "050007925f4f41788b9a6bc228db41b0"
I20240502 07:59:18.013749 30827 raft_consensus.cc:3032] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:18.013820 30827 raft_consensus.cc:729] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Running, Role: LEADER
I20240502 07:59:18.013968 30827 consensus_queue.cc:260] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } }
I20240502 07:59:18.014098 30827 raft_consensus.cc:3037] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:18.014961 30827 raft_consensus.cc:2454] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b205acd80ed40cdad98e381fb296499 in term 2.
I20240502 07:59:18.015297 31548 leader_election.cc:302] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 050007925f4f41788b9a6bc228db41b0, 9b205acd80ed40cdad98e381fb296499; no voters:
I20240502 07:59:18.015492 31914 raft_consensus.cc:2781] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:18.015602 31914 raft_consensus.cc:686] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [term 2 LEADER]: Becoming Leader. State: Replica: 9b205acd80ed40cdad98e381fb296499, State: Running, Role: LEADER
I20240502 07:59:18.015749 31914 consensus_queue.cc:237] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } }
I20240502 07:59:18.017544 30681 catalog_manager.cc:5484] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 reported cstate change: term changed from 1 to 2, leader changed from 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) to 9b205acd80ed40cdad98e381fb296499 (127.18.13.132). New cstate: current_term: 2 leader_uuid: "9b205acd80ed40cdad98e381fb296499" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:18.026121 30827 raft_consensus.cc:1261] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Refusing update from remote peer 9b205acd80ed40cdad98e381fb296499: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 07:59:18.026648 32225 consensus_queue.cc:1035] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [LEADER]: Connected to new peer: Peer: permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:18.134630 30958 raft_consensus.cc:1261] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Refusing update from remote peer c6642c4ab0104a9da2b204f9e965abfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 07:59:18.135367 31911 consensus_queue.cc:1035] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:18.187242 31614 consensus_queue.cc:237] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } }
I20240502 07:59:18.187934 32225 raft_consensus.cc:2932] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } } }
I20240502 07:59:18.189311 30669 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7b9fbec1717044a5839d0d0f2c967f6f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 07:59:18.190258 30681 catalog_manager.cc:5484] T 7b9fbec1717044a5839d0d0f2c967f6f P 9b205acd80ed40cdad98e381fb296499 reported cstate change: config changed from index 4 to 6, VOTER 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) evicted. New cstate: current_term: 2 leader_uuid: "9b205acd80ed40cdad98e381fb296499" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:18.192809 30807 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9fbec1717044a5839d0d0f2c967f6f with delete_type TABLET_DATA_TOMBSTONED (TS 050007925f4f41788b9a6bc228db41b0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45650
I20240502 07:59:18.193265 32315 tablet_replica.cc:330] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0: stopping tablet replica
I20240502 07:59:18.193524 32315 raft_consensus.cc:2229] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:18.193650 32315 raft_consensus.cc:2258] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:18.194447 32315 ts_tablet_manager.cc:1875] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:18.196208 32315 ts_tablet_manager.cc:1888] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 07:59:18.196290 32315 log.cc:1196] T 7b9fbec1717044a5839d0d0f2c967f6f P 050007925f4f41788b9a6bc228db41b0: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/7b9fbec1717044a5839d0d0f2c967f6f
I20240502 07:59:18.197173 30669 catalog_manager.cc:4830] TS 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133): tablet 7b9fbec1717044a5839d0d0f2c967f6f (table rebalance_test_table_2 [id=fa7ed001f0f5412481ea3f02e7239eb3]) successfully deleted
I20240502 07:59:18.256306 30958 raft_consensus.cc:1261] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Refusing update from remote peer 050007925f4f41788b9a6bc228db41b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 07:59:18.256795 32007 consensus_queue.cc:1035] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:18.257725 30827 raft_consensus.cc:1261] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Refusing update from remote peer c6642c4ab0104a9da2b204f9e965abfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 07:59:18.258144 32228 consensus_queue.cc:1035] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:18.272341 30958 raft_consensus.cc:1261] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Refusing update from remote peer 9b205acd80ed40cdad98e381fb296499: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 07:59:18.272737 32225 consensus_queue.cc:1035] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:18.274428 30827 raft_consensus.cc:1261] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Refusing update from remote peer c6642c4ab0104a9da2b204f9e965abfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 07:59:18.274848 31911 consensus_queue.cc:1035] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:18.288679 31483 consensus_queue.cc:237] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } }
I20240502 07:59:18.289294 31911 raft_consensus.cc:2932] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } } }
I20240502 07:59:18.291208 30670 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39209b4f0a1444b6b80a155657217fe8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 07:59:18.292047 30681 catalog_manager.cc:5484] T 39209b4f0a1444b6b80a155657217fe8 P c6642c4ab0104a9da2b204f9e965abfd reported cstate change: config changed from index 3 to 5, VOTER 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) evicted. New cstate: current_term: 2 leader_uuid: "c6642c4ab0104a9da2b204f9e965abfd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:18.292744 30958 consensus_queue.cc:237] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: true } }
I20240502 07:59:18.293593 32010 raft_consensus.cc:2932] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: true } } }
W20240502 07:59:18.294319 30891 consensus_peers.cc:485] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 -> Peer 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133): Couldn't send request to peer 050007925f4f41788b9a6bc228db41b0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53ae7b8fde174b12948b0d439d38851f. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:18.294507 30807 tablet_service.cc:1514] Processing DeleteTablet for tablet 39209b4f0a1444b6b80a155657217fe8 with delete_type TABLET_DATA_TOMBSTONED (TS 050007925f4f41788b9a6bc228db41b0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45650
I20240502 07:59:18.294739 32315 tablet_replica.cc:330] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0: stopping tablet replica
I20240502 07:59:18.294883 32315 raft_consensus.cc:2229] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:18.295006 32315 raft_consensus.cc:2258] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:18.295554 32315 ts_tablet_manager.cc:1875] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:18.295979 30681 catalog_manager.cc:5484] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 reported cstate change: config changed from index -1 to 3, NON_VOTER 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) added. New cstate: current_term: 1 leader_uuid: "9f1fe3e3d9d34dd88971628d6018d833" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:18.297766 32315 ts_tablet_manager.cc:1888] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 07:59:18.297845 32315 log.cc:1196] T 39209b4f0a1444b6b80a155657217fe8 P 050007925f4f41788b9a6bc228db41b0: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/39209b4f0a1444b6b80a155657217fe8
I20240502 07:59:18.298317 30669 catalog_manager.cc:4830] TS 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133): tablet 39209b4f0a1444b6b80a155657217fe8 (table rebalance_test_table_0 [id=ead4dfe17a8246e8b14dd665edd372ca]) successfully deleted
I20240502 07:59:18.374734 30827 consensus_queue.cc:237] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } }
I20240502 07:59:18.375367 31779 raft_consensus.cc:2932] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } } }
I20240502 07:59:18.376633 30669 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4d43114a3f804c84b451505f967abe5c with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 07:59:18.377552 30681 catalog_manager.cc:5484] T 4d43114a3f804c84b451505f967abe5c P 050007925f4f41788b9a6bc228db41b0 reported cstate change: config changed from index 4 to 6, VOTER 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130) evicted. New cstate: current_term: 2 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:18.379768 30938 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d43114a3f804c84b451505f967abe5c with delete_type TABLET_DATA_TOMBSTONED (TS 9f1fe3e3d9d34dd88971628d6018d833 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47400
I20240502 07:59:18.380246 31483 consensus_queue.cc:237] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } }
I20240502 07:59:18.380347 32427 tablet_replica.cc:330] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833: stopping tablet replica
I20240502 07:59:18.380650 32427 raft_consensus.cc:2229] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:18.380816 32427 raft_consensus.cc:2258] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:18.380821 31911 raft_consensus.cc:2932] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } } }
I20240502 07:59:18.381906 32427 ts_tablet_manager.cc:1875] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:18.382221 30670 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 01800acde3ab4f5e849089ad6a0cb645 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 07:59:18.382968 30681 catalog_manager.cc:5484] T 01800acde3ab4f5e849089ad6a0cb645 P c6642c4ab0104a9da2b204f9e965abfd reported cstate change: config changed from index 3 to 5, VOTER 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130) evicted. New cstate: current_term: 2 leader_uuid: "c6642c4ab0104a9da2b204f9e965abfd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:18.384023 32427 ts_tablet_manager.cc:1888] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 07:59:18.384116 32427 log.cc:1196] T 4d43114a3f804c84b451505f967abe5c P 9f1fe3e3d9d34dd88971628d6018d833: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/4d43114a3f804c84b451505f967abe5c
I20240502 07:59:18.384732 30670 catalog_manager.cc:4830] TS 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130:40385): tablet 4d43114a3f804c84b451505f967abe5c (table rebalance_test_table_1 [id=b64ba5c7c71945868fde0618a6d5f280]) successfully deleted
I20240502 07:59:18.385203 30938 tablet_service.cc:1514] Processing DeleteTablet for tablet 01800acde3ab4f5e849089ad6a0cb645 with delete_type TABLET_DATA_TOMBSTONED (TS 9f1fe3e3d9d34dd88971628d6018d833 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47400
I20240502 07:59:18.385465 32427 tablet_replica.cc:330] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833: stopping tablet replica
I20240502 07:59:18.385588 32427 raft_consensus.cc:2229] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:18.385692 32427 raft_consensus.cc:2258] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:18.386127 32427 ts_tablet_manager.cc:1875] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:18.387554 32427 ts_tablet_manager.cc:1888] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 07:59:18.387624 32427 log.cc:1196] T 01800acde3ab4f5e849089ad6a0cb645 P 9f1fe3e3d9d34dd88971628d6018d833: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/01800acde3ab4f5e849089ad6a0cb645
I20240502 07:59:18.388087 30670 catalog_manager.cc:4830] TS 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130:40385): tablet 01800acde3ab4f5e849089ad6a0cb645 (table rebalance_test_table_3 [id=e04c56fdd22b4f57bcb56d168d3ed16a]) successfully deleted
I20240502 07:59:18.418373 31614 consensus_queue.cc:237] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } }
I20240502 07:59:18.419054 31914 raft_consensus.cc:2932] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } } }
I20240502 07:59:18.420459 30669 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aa53770eb24c49938a6b0f56bdb85d24 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 07:59:18.421305 30681 catalog_manager.cc:5484] T aa53770eb24c49938a6b0f56bdb85d24 P 9b205acd80ed40cdad98e381fb296499 reported cstate change: config changed from index 3 to 5, VOTER 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130) evicted. New cstate: current_term: 2 leader_uuid: "9b205acd80ed40cdad98e381fb296499" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:18.423529 30938 tablet_service.cc:1514] Processing DeleteTablet for tablet aa53770eb24c49938a6b0f56bdb85d24 with delete_type TABLET_DATA_TOMBSTONED (TS 9f1fe3e3d9d34dd88971628d6018d833 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47400
I20240502 07:59:18.423760 32427 tablet_replica.cc:330] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833: stopping tablet replica
I20240502 07:59:18.423938 32427 raft_consensus.cc:2229] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:18.424054 32427 raft_consensus.cc:2258] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:18.424587 32427 ts_tablet_manager.cc:1875] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:18.426190 32427 ts_tablet_manager.cc:1888] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 07:59:18.426255 32427 log.cc:1196] T aa53770eb24c49938a6b0f56bdb85d24 P 9f1fe3e3d9d34dd88971628d6018d833: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/aa53770eb24c49938a6b0f56bdb85d24
I20240502 07:59:18.426728 30670 catalog_manager.cc:4830] TS 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130:40385): tablet aa53770eb24c49938a6b0f56bdb85d24 (table rebalance_test_table_1 [id=b64ba5c7c71945868fde0618a6d5f280]) successfully deleted
I20240502 07:59:18.435076 31483 consensus_queue.cc:237] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } }
I20240502 07:59:18.435660 31911 raft_consensus.cc:2932] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } } }
I20240502 07:59:18.436789 30670 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet de57879c802a4ac08fa6d96efed7360a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 07:59:18.437690 30681 catalog_manager.cc:5484] T de57879c802a4ac08fa6d96efed7360a P c6642c4ab0104a9da2b204f9e965abfd reported cstate change: config changed from index 3 to 5, VOTER 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) evicted. New cstate: current_term: 2 leader_uuid: "c6642c4ab0104a9da2b204f9e965abfd" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c6642c4ab0104a9da2b204f9e965abfd" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 38213 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:18.439853 30807 tablet_service.cc:1514] Processing DeleteTablet for tablet de57879c802a4ac08fa6d96efed7360a with delete_type TABLET_DATA_TOMBSTONED (TS 050007925f4f41788b9a6bc228db41b0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45650
I20240502 07:59:18.440083 32315 tablet_replica.cc:330] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0: stopping tablet replica
I20240502 07:59:18.440239 32315 raft_consensus.cc:2229] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:18.440346 32315 raft_consensus.cc:2258] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:18.440852 32315 ts_tablet_manager.cc:1875] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:18.442425 32315 ts_tablet_manager.cc:1888] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 07:59:18.442493 32315 log.cc:1196] T de57879c802a4ac08fa6d96efed7360a P 050007925f4f41788b9a6bc228db41b0: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/de57879c802a4ac08fa6d96efed7360a
I20240502 07:59:18.442957 30669 catalog_manager.cc:4830] TS 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133): tablet de57879c802a4ac08fa6d96efed7360a (table rebalance_test_table_2 [id=fa7ed001f0f5412481ea3f02e7239eb3]) successfully deleted
I20240502 07:59:18.527316 30827 raft_consensus.cc:1261] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Refusing update from remote peer 9b205acd80ed40cdad98e381fb296499: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 07:59:18.527819 31914 consensus_queue.cc:1035] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [LEADER]: Connected to new peer: Peer: permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:18.628329 31614 consensus_queue.cc:237] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } }
I20240502 07:59:18.629031 32225 raft_consensus.cc:2932] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } } }
I20240502 07:59:18.630419 30669 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3741c7c8018b412da5fd73fbbf052102 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 07:59:18.631282 30684 catalog_manager.cc:5484] T 3741c7c8018b412da5fd73fbbf052102 P 9b205acd80ed40cdad98e381fb296499 reported cstate change: config changed from index 4 to 6, VOTER 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) evicted. New cstate: current_term: 2 leader_uuid: "9b205acd80ed40cdad98e381fb296499" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9b205acd80ed40cdad98e381fb296499" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 38863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:18.633533 30807 tablet_service.cc:1514] Processing DeleteTablet for tablet 3741c7c8018b412da5fd73fbbf052102 with delete_type TABLET_DATA_TOMBSTONED (TS 050007925f4f41788b9a6bc228db41b0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:45650
I20240502 07:59:18.633767 32315 tablet_replica.cc:330] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0: stopping tablet replica
I20240502 07:59:18.633911 32315 raft_consensus.cc:2229] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:18.634019 32315 raft_consensus.cc:2258] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:18.634542 32315 ts_tablet_manager.cc:1875] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:18.636261 32315 ts_tablet_manager.cc:1888] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 07:59:18.636334 32315 log.cc:1196] T 3741c7c8018b412da5fd73fbbf052102 P 050007925f4f41788b9a6bc228db41b0: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/3741c7c8018b412da5fd73fbbf052102
I20240502 07:59:18.636811 30669 catalog_manager.cc:4830] TS 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133): tablet 3741c7c8018b412da5fd73fbbf052102 (table rebalance_test_table_0 [id=ead4dfe17a8246e8b14dd665edd372ca]) successfully deleted
I20240502 07:59:18.848892 32485 ts_tablet_manager.cc:902] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0: Initiating tablet copy from peer 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130:40385)
I20240502 07:59:18.849381 32485 tablet_copy_client.cc:308] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:40385
I20240502 07:59:18.849918 30978 tablet_copy_service.cc:140] P 9f1fe3e3d9d34dd88971628d6018d833: Received BeginTabletCopySession request for tablet 53ae7b8fde174b12948b0d439d38851f from peer 050007925f4f41788b9a6bc228db41b0 ({username='slave'} at 127.18.13.129:47267)
I20240502 07:59:18.850040 30978 tablet_copy_service.cc:161] P 9f1fe3e3d9d34dd88971628d6018d833: Beginning new tablet copy session on tablet 53ae7b8fde174b12948b0d439d38851f from peer 050007925f4f41788b9a6bc228db41b0 at {username='slave'} at 127.18.13.129:47267: session id = 050007925f4f41788b9a6bc228db41b0-53ae7b8fde174b12948b0d439d38851f
I20240502 07:59:18.850773 30978 tablet_copy_source_session.cc:205] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:18.851706 32485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53ae7b8fde174b12948b0d439d38851f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:18.853804 32485 tablet_copy_client.cc:783] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:18.853978 32485 tablet_copy_client.cc:647] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:18.855232 32485 tablet_copy_client.cc:523] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:18.856443 32485 tablet_bootstrap.cc:492] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0: Bootstrap starting.
I20240502 07:59:18.860518 32485 tablet_bootstrap.cc:492] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:18.860746 32485 tablet_bootstrap.cc:492] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0: Bootstrap complete.
I20240502 07:59:18.860901 32485 ts_tablet_manager.cc:1367] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 07:59:18.861349 32485 raft_consensus.cc:348] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: true } }
I20240502 07:59:18.861465 32485 raft_consensus.cc:729] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Initialized, Role: LEARNER
I20240502 07:59:18.861577 32485 consensus_queue.cc:260] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: true } }
I20240502 07:59:18.861855 32485 ts_tablet_manager.cc:1398] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:18.862443 30978 tablet_copy_service.cc:342] P 9f1fe3e3d9d34dd88971628d6018d833: Request end of tablet copy session 050007925f4f41788b9a6bc228db41b0-53ae7b8fde174b12948b0d439d38851f received from {username='slave'} at 127.18.13.129:47267
I20240502 07:59:18.862545 30978 tablet_copy_service.cc:433] P 9f1fe3e3d9d34dd88971628d6018d833: ending tablet copy session 050007925f4f41788b9a6bc228db41b0-53ae7b8fde174b12948b0d439d38851f on tablet 53ae7b8fde174b12948b0d439d38851f with peer 050007925f4f41788b9a6bc228db41b0
I20240502 07:59:19.292184 30827 raft_consensus.cc:1203] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240502 07:59:19.677095 32501 raft_consensus.cc:1050] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833: attempting to promote NON_VOTER 050007925f4f41788b9a6bc228db41b0 to VOTER
I20240502 07:59:19.677465 32501 consensus_queue.cc:237] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } }
I20240502 07:59:19.678578 30827 raft_consensus.cc:1261] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [term 1 LEARNER]: Refusing update from remote peer 9f1fe3e3d9d34dd88971628d6018d833: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:19.678975 32501 consensus_queue.cc:1035] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:19.680859 32502 raft_consensus.cc:2932] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } } }
I20240502 07:59:19.681319 30827 raft_consensus.cc:2932] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } } }
I20240502 07:59:19.683481 30681 catalog_manager.cc:5484] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 reported cstate change: config changed from index 3 to 4, 050007925f4f41788b9a6bc228db41b0 (127.18.13.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9f1fe3e3d9d34dd88971628d6018d833" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:19.875325 30958 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53ae7b8fde174b12948b0d439d38851f"
dest_uuid: "9f1fe3e3d9d34dd88971628d6018d833"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45018
I20240502 07:59:19.875458 30958 raft_consensus.cc:595] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:20.087642 30958 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53ae7b8fde174b12948b0d439d38851f"
dest_uuid: "9f1fe3e3d9d34dd88971628d6018d833"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45040
I20240502 07:59:20.087819 30958 raft_consensus.cc:595] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:20.120813 32484 raft_consensus.cc:979] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833: : Instructing follower 050007925f4f41788b9a6bc228db41b0 to start an election
I20240502 07:59:20.120967 32484 raft_consensus.cc:1067] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Signalling peer 050007925f4f41788b9a6bc228db41b0 to start an election
I20240502 07:59:20.121492 30827 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53ae7b8fde174b12948b0d439d38851f"
dest_uuid: "050007925f4f41788b9a6bc228db41b0"
from {username='slave'} at 127.18.13.130:52151
I20240502 07:59:20.121652 30827 raft_consensus.cc:482] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:20.121706 30827 raft_consensus.cc:3037] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:20.122536 30827 raft_consensus.cc:504] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } }
I20240502 07:59:20.122836 30827 leader_election.cc:288] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 2 election: Requested vote from peers 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130:40385)
I20240502 07:59:20.123335 30958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53ae7b8fde174b12948b0d439d38851f" candidate_uuid: "050007925f4f41788b9a6bc228db41b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "9f1fe3e3d9d34dd88971628d6018d833"
I20240502 07:59:20.123451 30958 raft_consensus.cc:3032] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:20.123489 30958 raft_consensus.cc:729] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9f1fe3e3d9d34dd88971628d6018d833, State: Running, Role: LEADER
I20240502 07:59:20.123616 30958 consensus_queue.cc:260] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } }
I20240502 07:59:20.123726 30958 raft_consensus.cc:3037] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:20.124519 30958 raft_consensus.cc:2454] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 050007925f4f41788b9a6bc228db41b0 in term 2.
I20240502 07:59:20.124827 30761 leader_election.cc:302] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 050007925f4f41788b9a6bc228db41b0, 9f1fe3e3d9d34dd88971628d6018d833; no voters:
I20240502 07:59:20.125010 31779 raft_consensus.cc:2781] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:20.125175 31779 raft_consensus.cc:686] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [term 2 LEADER]: Becoming Leader. State: Replica: 050007925f4f41788b9a6bc228db41b0, State: Running, Role: LEADER
I20240502 07:59:20.125344 31779 consensus_queue.cc:237] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } }
I20240502 07:59:20.126806 30684 catalog_manager.cc:5484] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 reported cstate change: term changed from 1 to 2, leader changed from 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130) to 050007925f4f41788b9a6bc228db41b0 (127.18.13.129). New cstate: current_term: 2 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 07:59:20.129423 30669 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 53ae7b8fde174b12948b0d439d38851f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 050007925f4f41788b9a6bc228db41b0 (127.18.13.129:40133) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 07:59:20.731096 30958 raft_consensus.cc:1261] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Refusing update from remote peer 050007925f4f41788b9a6bc228db41b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 07:59:20.731588 32556 consensus_queue.cc:1035] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1fe3e3d9d34dd88971628d6018d833" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 40385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:20.764253 30827 consensus_queue.cc:237] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } }
I20240502 07:59:20.765039 32559 raft_consensus.cc:2932] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } } }
I20240502 07:59:20.766546 30669 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53ae7b8fde174b12948b0d439d38851f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 07:59:20.767627 30681 catalog_manager.cc:5484] T 53ae7b8fde174b12948b0d439d38851f P 050007925f4f41788b9a6bc228db41b0 reported cstate change: config changed from index 4 to 6, VOTER 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130) evicted. New cstate: current_term: 2 leader_uuid: "050007925f4f41788b9a6bc228db41b0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "050007925f4f41788b9a6bc228db41b0" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 40133 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:20.770650 30938 tablet_service.cc:1514] Processing DeleteTablet for tablet 53ae7b8fde174b12948b0d439d38851f with delete_type TABLET_DATA_TOMBSTONED (TS 9f1fe3e3d9d34dd88971628d6018d833 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47400
I20240502 07:59:20.770987 32565 tablet_replica.cc:330] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833: stopping tablet replica
I20240502 07:59:20.771118 32565 raft_consensus.cc:2229] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:20.771230 32565 raft_consensus.cc:2258] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:20.771775 32565 ts_tablet_manager.cc:1875] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:20.773797 32565 ts_tablet_manager.cc:1888] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 07:59:20.773877 32565 log.cc:1196] T 53ae7b8fde174b12948b0d439d38851f P 9f1fe3e3d9d34dd88971628d6018d833: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/53ae7b8fde174b12948b0d439d38851f
I20240502 07:59:20.774498 30670 catalog_manager.cc:4830] TS 9f1fe3e3d9d34dd88971628d6018d833 (127.18.13.130:40385): tablet 53ae7b8fde174b12948b0d439d38851f (table rebalance_test_table_3 [id=e04c56fdd22b4f57bcb56d168d3ed16a]) successfully deleted
I20240502 07:59:20.971657 30938 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:20.973218 30807 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:20.973703 31595 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:20.975773 31464 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:21.063817 31595 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:21.064548 30807 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:21.065697 30938 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:21.066728 31464 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:21.160972 31595 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:21.161815 30938 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:21.161870 30807 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:21.162397 31464 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
53d352a2e03b4fce85db44fac169c811 | 127.18.13.190:34323 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.13.148:36333 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
050007925f4f41788b9a6bc228db41b0 | 127.18.13.129:40133 | HEALTHY | <none> | 4 | 0
9b205acd80ed40cdad98e381fb296499 | 127.18.13.132:38863 | HEALTHY | <none> | 3 | 0
9f1fe3e3d9d34dd88971628d6018d833 | 127.18.13.130:40385 | HEALTHY | <none> | 3 | 0
c6642c4ab0104a9da2b204f9e965abfd | 127.18.13.131:38213 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.18.13.129 | experimental | 127.18.13.129:40133
local_ip_for_outbound_sockets | 127.18.13.130 | experimental | 127.18.13.130:40385
local_ip_for_outbound_sockets | 127.18.13.131 | experimental | 127.18.13.131:38213
local_ip_for_outbound_sockets | 127.18.13.132 | experimental | 127.18.13.132:38863
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.18.13.129:40133
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.18.13.130:40385
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.18.13.131:38213
server_dump_info_path | /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.18.13.132:38863
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.18.13.148:36333 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 3
First Quartile | 3
Median | 3
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 07:59:21.190816 18486 log_verifier.cc:126] Checking tablet 00800e0317b8402a864b8df5ce3c425a
I20240502 07:59:21.198753 18486 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 00800e0317b8402a864b8df5ce3c425a
I20240502 07:59:21.198817 18486 log_verifier.cc:126] Checking tablet 01800acde3ab4f5e849089ad6a0cb645
I20240502 07:59:21.206167 18486 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 01800acde3ab4f5e849089ad6a0cb645
I20240502 07:59:21.206244 18486 log_verifier.cc:126] Checking tablet 3741c7c8018b412da5fd73fbbf052102
I20240502 07:59:21.213667 18486 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3741c7c8018b412da5fd73fbbf052102
I20240502 07:59:21.213742 18486 log_verifier.cc:126] Checking tablet 39209b4f0a1444b6b80a155657217fe8
I20240502 07:59:21.221151 18486 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 39209b4f0a1444b6b80a155657217fe8
I20240502 07:59:21.221223 18486 log_verifier.cc:126] Checking tablet 4d43114a3f804c84b451505f967abe5c
I20240502 07:59:21.228868 18486 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4d43114a3f804c84b451505f967abe5c
I20240502 07:59:21.228948 18486 log_verifier.cc:126] Checking tablet 53ae7b8fde174b12948b0d439d38851f
I20240502 07:59:21.236600 18486 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 53ae7b8fde174b12948b0d439d38851f
I20240502 07:59:21.236665 18486 log_verifier.cc:126] Checking tablet 584032d120a34aa4821240f2eeb7e96c
I20240502 07:59:21.243750 18486 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 584032d120a34aa4821240f2eeb7e96c
I20240502 07:59:21.243866 18486 log_verifier.cc:126] Checking tablet 5b3a9771d3a9438d9dbff501500386ef
I20240502 07:59:21.250900 18486 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 5b3a9771d3a9438d9dbff501500386ef
I20240502 07:59:21.250968 18486 log_verifier.cc:126] Checking tablet 6fdbdb698ee64caf9e478368aba84e60
I20240502 07:59:21.258090 18486 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 6fdbdb698ee64caf9e478368aba84e60
I20240502 07:59:21.258158 18486 log_verifier.cc:126] Checking tablet 7b9fbec1717044a5839d0d0f2c967f6f
I20240502 07:59:21.265717 18486 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 7b9fbec1717044a5839d0d0f2c967f6f
I20240502 07:59:21.265797 18486 log_verifier.cc:126] Checking tablet aa53770eb24c49938a6b0f56bdb85d24
I20240502 07:59:21.273260 18486 log_verifier.cc:177] Verified matching terms for 5 ops in tablet aa53770eb24c49938a6b0f56bdb85d24
I20240502 07:59:21.273334 18486 log_verifier.cc:126] Checking tablet c25e35fdef7649a0abd4d95259c11577
I20240502 07:59:21.280550 18486 log_verifier.cc:177] Verified matching terms for 1 ops in tablet c25e35fdef7649a0abd4d95259c11577
I20240502 07:59:21.280616 18486 log_verifier.cc:126] Checking tablet de57879c802a4ac08fa6d96efed7360a
I20240502 07:59:21.287902 18486 log_verifier.cc:177] Verified matching terms for 5 ops in tablet de57879c802a4ac08fa6d96efed7360a
I20240502 07:59:21.288592 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 30745
I20240502 07:59:21.295392 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 30876
I20240502 07:59:21.301548 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 31402
I20240502 07:59:21.307217 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 31533
I20240502 07:59:21.312949 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 30654
2024-05-02T07:59:21Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1 (12072 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (12073 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3
I20240502 07:59:21.347221 18486 test_util.cc:268] Using random seed: -82561905
I20240502 07:59:21.348062 18486 ts_itest-base.cc:112] Starting cluster with:
I20240502 07:59:21.348099 18486 ts_itest-base.cc:113] --------------
I20240502 07:59:21.348119 18486 ts_itest-base.cc:114] 6 tablet servers
I20240502 07:59:21.348135 18486 ts_itest-base.cc:115] 3 replicas per TS
I20240502 07:59:21.348150 18486 ts_itest-base.cc:116] --------------
2024-05-02T07:59:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T07:59:21Z Disabled control of system clock
I20240502 07:59:21.356559 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.13.190:43401
--webserver_interface=127.18.13.190
--webserver_port=0
--builtin_ntp_servers=127.18.13.148:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.13.190:43401
--location_mapping_cmd=/tmp/dist-test-taskeLuhys/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 07:59:21.464424 32666 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 07:59:21.464711 32666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:21.464772 32666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:21.464834 32666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:21.467911 32666 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:21.467988 32666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 07:59:21.468022 32666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:21.468065 32666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 07:59:21.468106 32666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 07:59:21.471877 32666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskeLuhys/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--ipki_ca_key_size=768
--master_addresses=127.18.13.190:43401
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.13.190:43401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.18.13.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.32666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:21.472854 32666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:21.473995 32666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:21.481884 32671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:21.482450 32666 server_base.cc:1034] running on GCE node
W20240502 07:59:21.482964 32674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:21.483206 32672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:21.483750 32666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:21.484683 32666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:21.485837 32666 hybrid_clock.cc:648] HybridClock initialized: now 1714636761485808 us; error 45 us; skew 500 ppm
I20240502 07:59:21.487679 32666 webserver.cc:460] Webserver started at http://127.18.13.190:36129/ using document root <none> and password file <none>
I20240502 07:59:21.488257 32666 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:21.488312 32666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:21.488500 32666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:21.490276 32666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c42bd24b1bb94f6b932064ce205cce77"
format_stamp: "Formatted at 2024-05-02 07:59:21 on dist-test-slave-f46z"
I20240502 07:59:21.490744 32666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c42bd24b1bb94f6b932064ce205cce77"
format_stamp: "Formatted at 2024-05-02 07:59:21 on dist-test-slave-f46z"
I20240502 07:59:21.494022 32666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:21.496152 32680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:21.497148 32666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:21.497267 32666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c42bd24b1bb94f6b932064ce205cce77"
format_stamp: "Formatted at 2024-05-02 07:59:21 on dist-test-slave-f46z"
I20240502 07:59:21.497350 32666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:21.525068 32666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:21.525875 32666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:21.526075 32666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:21.533077 32666 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.190:43401
I20240502 07:59:21.533082 32732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.190:43401 every 8 connection(s)
I20240502 07:59:21.534126 32666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 07:59:21.536687 32733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:21.541671 32733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77: Bootstrap starting.
I20240502 07:59:21.543020 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 32666
I20240502 07:59:21.543177 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 07:59:21.544658 32733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:21.545418 32733 log.cc:824] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:21.547111 32733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77: No bootstrap required, opened a new log
I20240502 07:59:21.549942 32733 raft_consensus.cc:348] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42bd24b1bb94f6b932064ce205cce77" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 43401 } }
I20240502 07:59:21.550137 32733 raft_consensus.cc:374] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:21.550210 32733 raft_consensus.cc:729] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42bd24b1bb94f6b932064ce205cce77, State: Initialized, Role: FOLLOWER
I20240502 07:59:21.550688 32733 consensus_queue.cc:260] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42bd24b1bb94f6b932064ce205cce77" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 43401 } }
I20240502 07:59:21.550839 32733 raft_consensus.cc:388] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:21.550900 32733 raft_consensus.cc:482] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:21.551074 32733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:21.552017 32733 raft_consensus.cc:504] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42bd24b1bb94f6b932064ce205cce77" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 43401 } }
I20240502 07:59:21.552392 32733 leader_election.cc:302] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c42bd24b1bb94f6b932064ce205cce77; no voters:
I20240502 07:59:21.552704 32733 leader_election.cc:288] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:21.552789 32738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:21.553381 32738 raft_consensus.cc:686] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [term 1 LEADER]: Becoming Leader. State: Replica: c42bd24b1bb94f6b932064ce205cce77, State: Running, Role: LEADER
I20240502 07:59:21.553658 32738 consensus_queue.cc:237] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42bd24b1bb94f6b932064ce205cce77" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 43401 } }
I20240502 07:59:21.553805 32733 sys_catalog.cc:564] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:59:21.554931 32739 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c42bd24b1bb94f6b932064ce205cce77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42bd24b1bb94f6b932064ce205cce77" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 43401 } } }
I20240502 07:59:21.555030 32739 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:21.555303 32740 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c42bd24b1bb94f6b932064ce205cce77. Latest consensus state: current_term: 1 leader_uuid: "c42bd24b1bb94f6b932064ce205cce77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42bd24b1bb94f6b932064ce205cce77" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 43401 } } }
I20240502 07:59:21.555385 32740 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77 [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:21.555480 32747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:59:21.557977 32747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:59:21.563022 32747 catalog_manager.cc:1347] Generated new cluster ID: d8fccacd35d7429dbae5308298ec8bc5
I20240502 07:59:21.563094 32747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:59:21.572158 32747 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:59:21.573261 32747 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:59:21.579416 32747 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c42bd24b1bb94f6b932064ce205cce77: Generated new TSK 0
I20240502 07:59:21.580277 32747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:59:21.591989 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.129:0
--local_ip_for_outbound_sockets=127.18.13.129
--webserver_interface=127.18.13.129
--webserver_port=0
--tserver_master_addrs=127.18.13.190:43401
--builtin_ntp_servers=127.18.13.148:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 07:59:21.709034 32757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:21.709280 32757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:21.709349 32757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:21.712416 32757 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:21.712492 32757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:21.712580 32757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.129
I20240502 07:59:21.716192 32757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.18.13.129
--webserver_port=0
--tserver_master_addrs=127.18.13.190:43401
--never_fsync=true
--heap_profile_path=/tmp/kudu.32757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.129
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:21.717159 32757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:21.718338 32757 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:21.726281 32763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:21.726986 32762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:21.727761 32765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:21.728427 32757 server_base.cc:1034] running on GCE node
I20240502 07:59:21.728823 32757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:21.729472 32757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:21.730648 32757 hybrid_clock.cc:648] HybridClock initialized: now 1714636761730632 us; error 35 us; skew 500 ppm
I20240502 07:59:21.737349 32757 webserver.cc:460] Webserver started at http://127.18.13.129:40497/ using document root <none> and password file <none>
I20240502 07:59:21.737977 32757 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:21.738044 32757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:21.738245 32757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:21.740132 32757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2ca73afa63a04c378a0948e9691601fd"
format_stamp: "Formatted at 2024-05-02 07:59:21 on dist-test-slave-f46z"
I20240502 07:59:21.740657 32757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2ca73afa63a04c378a0948e9691601fd"
format_stamp: "Formatted at 2024-05-02 07:59:21 on dist-test-slave-f46z"
I20240502 07:59:21.744254 32757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 07:59:21.746654 303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:21.747735 32757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:21.747895 32757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2ca73afa63a04c378a0948e9691601fd"
format_stamp: "Formatted at 2024-05-02 07:59:21 on dist-test-slave-f46z"
I20240502 07:59:21.747988 32757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:21.760366 32757 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:21.760982 32757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:21.761137 32757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:21.761652 32757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:21.762512 32757 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:21.762557 32757 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:21.762602 32757 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:21.762632 32757 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:21.771044 32757 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.129:46429
I20240502 07:59:21.771080 416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.129:46429 every 8 connection(s)
I20240502 07:59:21.772039 32757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 07:59:21.778055 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 32757
I20240502 07:59:21.778193 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 07:59:21.780521 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.130:0
--local_ip_for_outbound_sockets=127.18.13.130
--webserver_interface=127.18.13.130
--webserver_port=0
--tserver_master_addrs=127.18.13.190:43401
--builtin_ntp_servers=127.18.13.148:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:21.781155 417 heartbeater.cc:347] Connected to a master server at 127.18.13.190:43401
I20240502 07:59:21.781399 417 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:21.782053 417 heartbeater.cc:511] Master 127.18.13.190:43401 requested a full tablet report, sending...
I20240502 07:59:21.818605 32697 ts_manager.cc:194] Registered new tserver with Master: 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429)
I20240502 07:59:21.820529 32697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.129:59213
W20240502 07:59:21.888738 420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:21.888975 420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:21.889021 420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:21.892030 420 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:21.892092 420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:21.892159 420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.130
I20240502 07:59:21.895737 420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.18.13.130
--webserver_port=0
--tserver_master_addrs=127.18.13.190:43401
--never_fsync=true
--heap_profile_path=/tmp/kudu.420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.130
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:21.896612 420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:21.897722 420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:21.906286 429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:21.906389 427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:21.907135 426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:21.907766 420 server_base.cc:1034] running on GCE node
I20240502 07:59:21.908159 420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:21.908779 420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:21.909926 420 hybrid_clock.cc:648] HybridClock initialized: now 1714636761909916 us; error 28 us; skew 500 ppm
I20240502 07:59:21.911659 420 webserver.cc:460] Webserver started at http://127.18.13.130:33841/ using document root <none> and password file <none>
I20240502 07:59:21.912235 420 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:21.912307 420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:21.912500 420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:21.914099 420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "5a60a9fab026458293b88d08fb10e88a"
format_stamp: "Formatted at 2024-05-02 07:59:21 on dist-test-slave-f46z"
I20240502 07:59:21.914559 420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "5a60a9fab026458293b88d08fb10e88a"
format_stamp: "Formatted at 2024-05-02 07:59:21 on dist-test-slave-f46z"
I20240502 07:59:21.917851 420 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:21.920003 435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:21.920986 420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:21.921092 420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "5a60a9fab026458293b88d08fb10e88a"
format_stamp: "Formatted at 2024-05-02 07:59:21 on dist-test-slave-f46z"
I20240502 07:59:21.921175 420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:21.951915 420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:21.952592 420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:21.952755 420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:21.953294 420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:21.954196 420 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:21.954247 420 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:21.954317 420 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:21.954344 420 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:21.963613 420 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.130:37457
I20240502 07:59:21.963637 548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.130:37457 every 8 connection(s)
I20240502 07:59:21.964643 420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 07:59:21.966620 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 420
I20240502 07:59:21.966774 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 07:59:21.969153 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.131:0
--local_ip_for_outbound_sockets=127.18.13.131
--webserver_interface=127.18.13.131
--webserver_port=0
--tserver_master_addrs=127.18.13.190:43401
--builtin_ntp_servers=127.18.13.148:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:21.974467 549 heartbeater.cc:347] Connected to a master server at 127.18.13.190:43401
I20240502 07:59:21.974694 549 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:21.975240 549 heartbeater.cc:511] Master 127.18.13.190:43401 requested a full tablet report, sending...
I20240502 07:59:22.021817 32697 ts_manager.cc:194] Registered new tserver with Master: 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457)
I20240502 07:59:22.022550 32697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.130:42489
W20240502 07:59:22.094194 552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:22.094451 552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:22.094504 552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:22.097710 552 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:22.097796 552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:22.097891 552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.131
I20240502 07:59:22.101778 552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.18.13.131
--webserver_port=0
--tserver_master_addrs=127.18.13.190:43401
--never_fsync=true
--heap_profile_path=/tmp/kudu.552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.131
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:22.102849 552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:22.104094 552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:22.112831 558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:22.112927 559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:22.113734 561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:22.114359 552 server_base.cc:1034] running on GCE node
I20240502 07:59:22.114876 552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:22.115644 552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:22.116816 552 hybrid_clock.cc:648] HybridClock initialized: now 1714636762116807 us; error 66 us; skew 500 ppm
I20240502 07:59:22.118755 552 webserver.cc:460] Webserver started at http://127.18.13.131:37159/ using document root <none> and password file <none>
I20240502 07:59:22.119349 552 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:22.119413 552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:22.119601 552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:22.121400 552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "733b523e2f3e43539b480b034726d7ec"
format_stamp: "Formatted at 2024-05-02 07:59:22 on dist-test-slave-f46z"
I20240502 07:59:22.121889 552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "733b523e2f3e43539b480b034726d7ec"
format_stamp: "Formatted at 2024-05-02 07:59:22 on dist-test-slave-f46z"
I20240502 07:59:22.125430 552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:22.127760 567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:22.128842 552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:22.128949 552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "733b523e2f3e43539b480b034726d7ec"
format_stamp: "Formatted at 2024-05-02 07:59:22 on dist-test-slave-f46z"
I20240502 07:59:22.129050 552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:22.147403 552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:22.148094 552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:22.148267 552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:22.148837 552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:22.149744 552 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:22.149791 552 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:22.149839 552 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:22.149876 552 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:22.158372 552 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.131:37189
I20240502 07:59:22.158404 680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.131:37189 every 8 connection(s)
I20240502 07:59:22.159344 552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 07:59:22.166409 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 552
I20240502 07:59:22.166555 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 07:59:22.168205 681 heartbeater.cc:347] Connected to a master server at 127.18.13.190:43401
I20240502 07:59:22.168457 681 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:22.169028 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.132:0
--local_ip_for_outbound_sockets=127.18.13.132
--webserver_interface=127.18.13.132
--webserver_port=0
--tserver_master_addrs=127.18.13.190:43401
--builtin_ntp_servers=127.18.13.148:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:22.169108 681 heartbeater.cc:511] Master 127.18.13.190:43401 requested a full tablet report, sending...
I20240502 07:59:22.203037 32697 ts_manager.cc:194] Registered new tserver with Master: 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:22.204012 32697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.131:35643
W20240502 07:59:22.278889 684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:22.279131 684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:22.279179 684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:22.282500 684 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:22.282581 684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:22.282651 684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.132
I20240502 07:59:22.286331 684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.18.13.132
--webserver_port=0
--tserver_master_addrs=127.18.13.190:43401
--never_fsync=true
--heap_profile_path=/tmp/kudu.684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.132
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:22.287238 684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:22.288410 684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:22.296720 693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:22.297430 690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:22.298120 691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:22.298861 684 server_base.cc:1034] running on GCE node
I20240502 07:59:22.299254 684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:22.299953 684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:22.301105 684 hybrid_clock.cc:648] HybridClock initialized: now 1714636762301081 us; error 41 us; skew 500 ppm
I20240502 07:59:22.302980 684 webserver.cc:460] Webserver started at http://127.18.13.132:44375/ using document root <none> and password file <none>
I20240502 07:59:22.303614 684 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:22.303680 684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:22.303910 684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:22.305589 684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "468fd6f1f0fb46ce87ab94f922e89661"
format_stamp: "Formatted at 2024-05-02 07:59:22 on dist-test-slave-f46z"
I20240502 07:59:22.306062 684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "468fd6f1f0fb46ce87ab94f922e89661"
format_stamp: "Formatted at 2024-05-02 07:59:22 on dist-test-slave-f46z"
I20240502 07:59:22.309412 684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:22.311576 699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:22.312606 684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:22.312717 684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "468fd6f1f0fb46ce87ab94f922e89661"
format_stamp: "Formatted at 2024-05-02 07:59:22 on dist-test-slave-f46z"
I20240502 07:59:22.312834 684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:22.324091 684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:22.324679 684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:22.324833 684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:22.325394 684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:22.326285 684 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:22.326336 684 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:22.326396 684 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:22.326426 684 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:22.334710 684 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.132:33885
I20240502 07:59:22.334736 812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.132:33885 every 8 connection(s)
I20240502 07:59:22.335615 684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:59:22.344503 813 heartbeater.cc:347] Connected to a master server at 127.18.13.190:43401
I20240502 07:59:22.344724 813 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:22.345242 813 heartbeater.cc:511] Master 127.18.13.190:43401 requested a full tablet report, sending...
I20240502 07:59:22.345424 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 684
I20240502 07:59:22.345541 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 07:59:22.349838 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.133:0
--local_ip_for_outbound_sockets=127.18.13.133
--webserver_interface=127.18.13.133
--webserver_port=0
--tserver_master_addrs=127.18.13.190:43401
--builtin_ntp_servers=127.18.13.148:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:22.392582 32697 ts_manager.cc:194] Registered new tserver with Master: 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132:33885)
I20240502 07:59:22.393309 32697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.132:44933
W20240502 07:59:22.472530 817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:22.472743 817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:22.472790 817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:22.475890 817 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:22.475956 817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:22.476049 817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.133
I20240502 07:59:22.479638 817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.18.13.133
--webserver_port=0
--tserver_master_addrs=127.18.13.190:43401
--never_fsync=true
--heap_profile_path=/tmp/kudu.817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.133
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:22.480579 817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:22.481703 817 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:22.489766 822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:22.490422 823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:22.491056 825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:22.491721 817 server_base.cc:1034] running on GCE node
I20240502 07:59:22.492153 817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:22.492792 817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:22.493942 817 hybrid_clock.cc:648] HybridClock initialized: now 1714636762493928 us; error 26 us; skew 500 ppm
I20240502 07:59:22.495635 817 webserver.cc:460] Webserver started at http://127.18.13.133:33513/ using document root <none> and password file <none>
I20240502 07:59:22.496189 817 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:22.496244 817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:22.496409 817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:22.497984 817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "857b0b25994641599b6e146cec879443"
format_stamp: "Formatted at 2024-05-02 07:59:22 on dist-test-slave-f46z"
I20240502 07:59:22.498423 817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "857b0b25994641599b6e146cec879443"
format_stamp: "Formatted at 2024-05-02 07:59:22 on dist-test-slave-f46z"
I20240502 07:59:22.501757 817 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:22.503901 831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:22.504875 817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 07:59:22.504967 817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "857b0b25994641599b6e146cec879443"
format_stamp: "Formatted at 2024-05-02 07:59:22 on dist-test-slave-f46z"
I20240502 07:59:22.505040 817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:22.521436 817 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:22.522032 817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:22.522188 817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:22.522699 817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:22.523572 817 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:22.523618 817 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:22.523658 817 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:22.523674 817 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:22.531770 817 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.133:44143
I20240502 07:59:22.531837 944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.133:44143 every 8 connection(s)
I20240502 07:59:22.532694 817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:59:22.537758 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 817
I20240502 07:59:22.537886 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 07:59:22.540238 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.134:0
--local_ip_for_outbound_sockets=127.18.13.134
--webserver_interface=127.18.13.134
--webserver_port=0
--tserver_master_addrs=127.18.13.190:43401
--builtin_ntp_servers=127.18.13.148:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:22.542342 945 heartbeater.cc:347] Connected to a master server at 127.18.13.190:43401
I20240502 07:59:22.542574 945 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:22.543143 945 heartbeater.cc:511] Master 127.18.13.190:43401 requested a full tablet report, sending...
I20240502 07:59:22.578207 32697 ts_manager.cc:194] Registered new tserver with Master: 857b0b25994641599b6e146cec879443 (127.18.13.133:44143)
I20240502 07:59:22.579121 32697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.133:56337
W20240502 07:59:22.652782 948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:22.653007 948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:22.653053 948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:22.656168 948 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:22.656230 948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:22.656297 948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.134
I20240502 07:59:22.659873 948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.18.13.134
--webserver_port=0
--tserver_master_addrs=127.18.13.190:43401
--never_fsync=true
--heap_profile_path=/tmp/kudu.948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.134
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:22.660770 948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:22.661839 948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:22.670316 957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:22.670405 954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:22.671087 955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:22.671746 948 server_base.cc:1034] running on GCE node
I20240502 07:59:22.672187 948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:22.672796 948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:22.673944 948 hybrid_clock.cc:648] HybridClock initialized: now 1714636762673935 us; error 29 us; skew 500 ppm
I20240502 07:59:22.675616 948 webserver.cc:460] Webserver started at http://127.18.13.134:41527/ using document root <none> and password file <none>
I20240502 07:59:22.676167 948 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:22.676230 948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:22.676405 948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:22.678022 948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9"
format_stamp: "Formatted at 2024-05-02 07:59:22 on dist-test-slave-f46z"
I20240502 07:59:22.678493 948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9"
format_stamp: "Formatted at 2024-05-02 07:59:22 on dist-test-slave-f46z"
I20240502 07:59:22.681743 948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:22.683925 963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:22.684911 948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:22.685016 948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9"
format_stamp: "Formatted at 2024-05-02 07:59:22 on dist-test-slave-f46z"
I20240502 07:59:22.685107 948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:22.708961 948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:22.709619 948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:22.709780 948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:22.710335 948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:22.711230 948 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:22.711294 948 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:22.711356 948 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:22.711381 948 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:22.719805 948 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.134:39939
I20240502 07:59:22.719856 1076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.134:39939 every 8 connection(s)
I20240502 07:59:22.720762 948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 07:59:22.726704 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 948
I20240502 07:59:22.726848 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 07:59:22.729878 1077 heartbeater.cc:347] Connected to a master server at 127.18.13.190:43401
I20240502 07:59:22.730113 1077 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:22.730648 1077 heartbeater.cc:511] Master 127.18.13.190:43401 requested a full tablet report, sending...
I20240502 07:59:22.766119 32697 ts_manager.cc:194] Registered new tserver with Master: 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134:39939)
I20240502 07:59:22.766804 32697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.134:53081
I20240502 07:59:22.777431 18486 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 07:59:22.786422 18486 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 07:59:22.786501 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 684
I20240502 07:59:22.792277 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 948
I20240502 07:59:22.797791 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 817
I20240502 07:59:22.823493 417 heartbeater.cc:503] Master 127.18.13.190:43401 was elected leader, sending a full tablet report...
I20240502 07:59:23.024925 549 heartbeater.cc:503] Master 127.18.13.190:43401 was elected leader, sending a full tablet report...
I20240502 07:59:23.206238 681 heartbeater.cc:503] Master 127.18.13.190:43401 was elected leader, sending a full tablet report...
I20240502 07:59:26.554644 32697 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46856:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 07:59:26.558007 32697 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 07:59:26.575129 483 tablet_service.cc:1467] Processing CreateTablet for tablet 42d23c677d594a6693a052ca2adc2de8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=442b95cfe4e84e99965954832dd504ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:26.575134 614 tablet_service.cc:1467] Processing CreateTablet for tablet b51370efd02a42ec920c531abf4834b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=442b95cfe4e84e99965954832dd504ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:26.575129 350 tablet_service.cc:1467] Processing CreateTablet for tablet b51370efd02a42ec920c531abf4834b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=442b95cfe4e84e99965954832dd504ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:26.575131 351 tablet_service.cc:1467] Processing CreateTablet for tablet 42d23c677d594a6693a052ca2adc2de8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=442b95cfe4e84e99965954832dd504ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:26.576179 351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d23c677d594a6693a052ca2adc2de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:26.576179 614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b51370efd02a42ec920c531abf4834b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:26.576179 483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d23c677d594a6693a052ca2adc2de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:26.576375 481 tablet_service.cc:1467] Processing CreateTablet for tablet cc751cb753a840d28fb8787ca1ecafb0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=442b95cfe4e84e99965954832dd504ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:26.576606 481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc751cb753a840d28fb8787ca1ecafb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:26.579345 349 tablet_service.cc:1467] Processing CreateTablet for tablet cc751cb753a840d28fb8787ca1ecafb0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=442b95cfe4e84e99965954832dd504ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:26.579600 349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc751cb753a840d28fb8787ca1ecafb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:26.579864 482 tablet_service.cc:1467] Processing CreateTablet for tablet b51370efd02a42ec920c531abf4834b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=442b95cfe4e84e99965954832dd504ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:26.580096 482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b51370efd02a42ec920c531abf4834b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:26.581280 613 tablet_service.cc:1467] Processing CreateTablet for tablet cc751cb753a840d28fb8787ca1ecafb0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=442b95cfe4e84e99965954832dd504ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:26.581532 613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc751cb753a840d28fb8787ca1ecafb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:26.582113 615 tablet_service.cc:1467] Processing CreateTablet for tablet 42d23c677d594a6693a052ca2adc2de8 (DEFAULT_TABLE table=rebalance_test_table_0 [id=442b95cfe4e84e99965954832dd504ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:26.582352 615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d23c677d594a6693a052ca2adc2de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:26.584810 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b51370efd02a42ec920c531abf4834b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:26.585467 1098 tablet_bootstrap.cc:492] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec: Bootstrap starting.
I20240502 07:59:26.587448 1096 tablet_bootstrap.cc:492] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a: Bootstrap starting.
I20240502 07:59:26.587996 1098 tablet_bootstrap.cc:654] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:26.588243 1097 tablet_bootstrap.cc:492] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd: Bootstrap starting.
I20240502 07:59:26.589226 1098 log.cc:824] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:26.589725 1096 tablet_bootstrap.cc:654] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:26.589892 1097 tablet_bootstrap.cc:654] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:26.590493 1097 log.cc:824] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:26.590487 1096 log.cc:824] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:26.591115 1098 tablet_bootstrap.cc:492] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec: No bootstrap required, opened a new log
I20240502 07:59:26.591295 1098 ts_tablet_manager.cc:1367] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 07:59:26.592084 1096 tablet_bootstrap.cc:492] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a: No bootstrap required, opened a new log
I20240502 07:59:26.592084 1097 tablet_bootstrap.cc:492] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd: No bootstrap required, opened a new log
I20240502 07:59:26.592253 1097 ts_tablet_manager.cc:1367] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 07:59:26.592253 1096 ts_tablet_manager.cc:1367] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 07:59:26.594203 1098 raft_consensus.cc:348] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } }
I20240502 07:59:26.594389 1098 raft_consensus.cc:374] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:26.594453 1098 raft_consensus.cc:729] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733b523e2f3e43539b480b034726d7ec, State: Initialized, Role: FOLLOWER
I20240502 07:59:26.594861 1098 consensus_queue.cc:260] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } }
I20240502 07:59:26.595674 1098 ts_tablet_manager.cc:1398] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 07:59:26.595969 1098 tablet_bootstrap.cc:492] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec: Bootstrap starting.
I20240502 07:59:26.596017 1096 raft_consensus.cc:348] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.596222 1096 raft_consensus.cc:374] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:26.596284 1096 raft_consensus.cc:729] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a60a9fab026458293b88d08fb10e88a, State: Initialized, Role: FOLLOWER
I20240502 07:59:26.596735 1096 consensus_queue.cc:260] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.597024 1098 tablet_bootstrap.cc:654] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:26.598096 1098 tablet_bootstrap.cc:492] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec: No bootstrap required, opened a new log
I20240502 07:59:26.598181 1098 ts_tablet_manager.cc:1367] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:26.598692 1096 ts_tablet_manager.cc:1398] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20240502 07:59:26.598733 1098 raft_consensus.cc:348] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.598842 1098 raft_consensus.cc:374] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:26.598876 1098 raft_consensus.cc:729] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733b523e2f3e43539b480b034726d7ec, State: Initialized, Role: FOLLOWER
I20240502 07:59:26.598937 1096 tablet_bootstrap.cc:492] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a: Bootstrap starting.
I20240502 07:59:26.599035 1098 consensus_queue.cc:260] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.599339 1098 ts_tablet_manager.cc:1398] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:26.599507 1098 tablet_bootstrap.cc:492] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec: Bootstrap starting.
I20240502 07:59:26.599956 1096 tablet_bootstrap.cc:654] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:26.600358 1098 tablet_bootstrap.cc:654] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:26.600919 1097 raft_consensus.cc:348] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } }
I20240502 07:59:26.601239 1097 raft_consensus.cc:374] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:26.601377 1097 raft_consensus.cc:729] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca73afa63a04c378a0948e9691601fd, State: Initialized, Role: FOLLOWER
I20240502 07:59:26.601832 1096 tablet_bootstrap.cc:492] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a: No bootstrap required, opened a new log
I20240502 07:59:26.601924 1096 ts_tablet_manager.cc:1367] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:26.602130 1097 consensus_queue.cc:260] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } }
I20240502 07:59:26.602403 1096 raft_consensus.cc:348] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } }
I20240502 07:59:26.602492 1096 raft_consensus.cc:374] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:26.602530 1096 raft_consensus.cc:729] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a60a9fab026458293b88d08fb10e88a, State: Initialized, Role: FOLLOWER
I20240502 07:59:26.602653 1096 consensus_queue.cc:260] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } }
I20240502 07:59:26.603132 1097 ts_tablet_manager.cc:1398] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240502 07:59:26.603247 1096 ts_tablet_manager.cc:1398] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:26.603397 1096 tablet_bootstrap.cc:492] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a: Bootstrap starting.
I20240502 07:59:26.603396 1098 tablet_bootstrap.cc:492] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec: No bootstrap required, opened a new log
I20240502 07:59:26.603479 1098 ts_tablet_manager.cc:1367] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 07:59:26.603996 1098 raft_consensus.cc:348] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.604107 1098 raft_consensus.cc:374] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:26.604144 1098 raft_consensus.cc:729] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733b523e2f3e43539b480b034726d7ec, State: Initialized, Role: FOLLOWER
I20240502 07:59:26.604288 1098 consensus_queue.cc:260] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.604525 1096 tablet_bootstrap.cc:654] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:26.604620 1098 ts_tablet_manager.cc:1398] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:26.605583 1097 tablet_bootstrap.cc:492] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: Bootstrap starting.
I20240502 07:59:26.605695 1096 tablet_bootstrap.cc:492] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a: No bootstrap required, opened a new log
I20240502 07:59:26.605794 1096 ts_tablet_manager.cc:1367] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 07:59:26.606254 1096 raft_consensus.cc:348] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.606356 1096 raft_consensus.cc:374] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:26.606389 1096 raft_consensus.cc:729] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a60a9fab026458293b88d08fb10e88a, State: Initialized, Role: FOLLOWER
I20240502 07:59:26.606438 1097 tablet_bootstrap.cc:654] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:26.606523 1096 consensus_queue.cc:260] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.606877 1096 ts_tablet_manager.cc:1398] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:26.607614 1097 tablet_bootstrap.cc:492] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: No bootstrap required, opened a new log
I20240502 07:59:26.607700 1097 ts_tablet_manager.cc:1367] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:26.608319 1097 raft_consensus.cc:348] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.608434 1097 raft_consensus.cc:374] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:26.608472 1097 raft_consensus.cc:729] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca73afa63a04c378a0948e9691601fd, State: Initialized, Role: FOLLOWER
I20240502 07:59:26.608606 1097 consensus_queue.cc:260] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.608940 1097 ts_tablet_manager.cc:1398] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:26.609119 1097 tablet_bootstrap.cc:492] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd: Bootstrap starting.
I20240502 07:59:26.609800 1097 tablet_bootstrap.cc:654] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:26.610780 1097 tablet_bootstrap.cc:492] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd: No bootstrap required, opened a new log
I20240502 07:59:26.610845 1097 ts_tablet_manager.cc:1367] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:59:26.611217 1097 raft_consensus.cc:348] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.611317 1097 raft_consensus.cc:374] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:26.611366 1097 raft_consensus.cc:729] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca73afa63a04c378a0948e9691601fd, State: Initialized, Role: FOLLOWER
I20240502 07:59:26.611498 1097 consensus_queue.cc:260] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.611737 1097 ts_tablet_manager.cc:1398] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:26.624193 1102 raft_consensus.cc:482] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:26.624393 1102 raft_consensus.cc:504] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } }
I20240502 07:59:26.625618 1102 leader_election.cc:288] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457), 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429)
I20240502 07:59:26.629618 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d23c677d594a6693a052ca2adc2de8" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca73afa63a04c378a0948e9691601fd" is_pre_election: true
I20240502 07:59:26.629689 503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d23c677d594a6693a052ca2adc2de8" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a" is_pre_election: true
I20240502 07:59:26.629885 371 raft_consensus.cc:2454] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 0.
I20240502 07:59:26.629953 503 raft_consensus.cc:2454] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 0.
I20240502 07:59:26.630293 568 leader_election.cc:302] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca73afa63a04c378a0948e9691601fd, 733b523e2f3e43539b480b034726d7ec; no voters:
I20240502 07:59:26.630499 1102 raft_consensus.cc:2781] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:26.630576 1102 raft_consensus.cc:482] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:26.630635 1102 raft_consensus.cc:3037] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:26.631701 1102 raft_consensus.cc:504] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } }
I20240502 07:59:26.632055 1102 leader_election.cc:288] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 election: Requested vote from peers 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457), 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429)
I20240502 07:59:26.632390 503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d23c677d594a6693a052ca2adc2de8" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a"
I20240502 07:59:26.632436 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d23c677d594a6693a052ca2adc2de8" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca73afa63a04c378a0948e9691601fd"
I20240502 07:59:26.632506 503 raft_consensus.cc:3037] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:26.632534 371 raft_consensus.cc:3037] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:26.633742 503 raft_consensus.cc:2454] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 1.
I20240502 07:59:26.633778 371 raft_consensus.cc:2454] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 1.
I20240502 07:59:26.634033 568 leader_election.cc:302] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca73afa63a04c378a0948e9691601fd, 733b523e2f3e43539b480b034726d7ec; no voters:
I20240502 07:59:26.634204 1102 raft_consensus.cc:2781] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:26.634411 1102 raft_consensus.cc:686] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Becoming Leader. State: Replica: 733b523e2f3e43539b480b034726d7ec, State: Running, Role: LEADER
I20240502 07:59:26.634733 1102 consensus_queue.cc:237] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } }
I20240502 07:59:26.637476 32695 catalog_manager.cc:5484] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 733b523e2f3e43539b480b034726d7ec (127.18.13.131). New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:26.663771 682 tablet.cc:2369] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:26.703119 1107 raft_consensus.cc:482] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:26.703264 1107 raft_consensus.cc:504] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.704411 1107 leader_election.cc:288] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457), 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:26.708194 503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51370efd02a42ec920c531abf4834b9" candidate_uuid: "2ca73afa63a04c378a0948e9691601fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a" is_pre_election: true
I20240502 07:59:26.708257 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51370efd02a42ec920c531abf4834b9" candidate_uuid: "2ca73afa63a04c378a0948e9691601fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733b523e2f3e43539b480b034726d7ec" is_pre_election: true
I20240502 07:59:26.708338 503 raft_consensus.cc:2454] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca73afa63a04c378a0948e9691601fd in term 0.
I20240502 07:59:26.708487 635 raft_consensus.cc:2454] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca73afa63a04c378a0948e9691601fd in term 0.
I20240502 07:59:26.708694 304 leader_election.cc:302] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca73afa63a04c378a0948e9691601fd, 5a60a9fab026458293b88d08fb10e88a; no voters:
I20240502 07:59:26.708946 1107 raft_consensus.cc:2781] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:26.709008 1107 raft_consensus.cc:482] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:26.709062 1107 raft_consensus.cc:3037] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:26.711174 1107 raft_consensus.cc:504] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.711639 1107 leader_election.cc:288] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [CANDIDATE]: Term 1 election: Requested vote from peers 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457), 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:26.712023 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51370efd02a42ec920c531abf4834b9" candidate_uuid: "2ca73afa63a04c378a0948e9691601fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733b523e2f3e43539b480b034726d7ec"
I20240502 07:59:26.712026 503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51370efd02a42ec920c531abf4834b9" candidate_uuid: "2ca73afa63a04c378a0948e9691601fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a"
I20240502 07:59:26.712126 503 raft_consensus.cc:3037] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:26.712133 635 raft_consensus.cc:3037] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:26.712867 503 raft_consensus.cc:2454] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca73afa63a04c378a0948e9691601fd in term 1.
I20240502 07:59:26.712880 635 raft_consensus.cc:2454] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca73afa63a04c378a0948e9691601fd in term 1.
I20240502 07:59:26.713126 304 leader_election.cc:302] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca73afa63a04c378a0948e9691601fd, 5a60a9fab026458293b88d08fb10e88a; no voters:
I20240502 07:59:26.713297 1107 raft_consensus.cc:2781] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:26.713636 1107 raft_consensus.cc:686] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEADER]: Becoming Leader. State: Replica: 2ca73afa63a04c378a0948e9691601fd, State: Running, Role: LEADER
I20240502 07:59:26.713994 1107 consensus_queue.cc:237] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.716997 32695 catalog_manager.cc:5484] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ca73afa63a04c378a0948e9691601fd (127.18.13.129). New cstate: current_term: 1 leader_uuid: "2ca73afa63a04c378a0948e9691601fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:26.719183 550 tablet.cc:2369] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:26.769894 1102 raft_consensus.cc:482] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:26.770032 1102 raft_consensus.cc:504] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.770443 1102 leader_election.cc:288] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429), 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457)
I20240502 07:59:26.770975 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc751cb753a840d28fb8787ca1ecafb0" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca73afa63a04c378a0948e9691601fd" is_pre_election: true
I20240502 07:59:26.771023 503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc751cb753a840d28fb8787ca1ecafb0" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a" is_pre_election: true
I20240502 07:59:26.771144 503 raft_consensus.cc:2454] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 0.
I20240502 07:59:26.771144 371 raft_consensus.cc:2454] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 0.
I20240502 07:59:26.771510 568 leader_election.cc:302] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a60a9fab026458293b88d08fb10e88a, 733b523e2f3e43539b480b034726d7ec; no voters:
I20240502 07:59:26.771716 1102 raft_consensus.cc:2781] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:26.771811 1102 raft_consensus.cc:482] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:26.771868 1102 raft_consensus.cc:3037] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:26.772784 1102 raft_consensus.cc:504] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:26.773160 1102 leader_election.cc:288] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 election: Requested vote from peers 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429), 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457)
I20240502 07:59:26.773545 503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc751cb753a840d28fb8787ca1ecafb0" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a"
I20240502 07:59:26.773545 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc751cb753a840d28fb8787ca1ecafb0" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca73afa63a04c378a0948e9691601fd"
I20240502 07:59:26.773653 371 raft_consensus.cc:3037] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:26.773680 503 raft_consensus.cc:3037] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:26.774408 503 raft_consensus.cc:2454] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 1.
I20240502 07:59:26.774561 371 raft_consensus.cc:2454] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 1.
I20240502 07:59:26.774708 568 leader_election.cc:302] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a60a9fab026458293b88d08fb10e88a, 733b523e2f3e43539b480b034726d7ec; no voters:
I20240502 07:59:26.774976 1102 raft_consensus.cc:2781] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:26.775095 1102 raft_consensus.cc:686] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Becoming Leader. State: Replica: 733b523e2f3e43539b480b034726d7ec, State: Running, Role: LEADER
I20240502 07:59:26.775236 1102 consensus_queue.cc:237] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
W20240502 07:59:26.777020 418 tablet.cc:2369] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:26.776997 32695 catalog_manager.cc:5484] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 733b523e2f3e43539b480b034726d7ec (127.18.13.131). New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } }
W20240502 07:59:26.918569 1120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:26.918834 1120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:26.921968 1120 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 07:59:27.023138 635 raft_consensus.cc:1261] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:27.024060 503 raft_consensus.cc:1261] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:27.025049 502 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:27.025714 371 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:27.026047 1117 consensus_queue.cc:1035] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:27.025717 370 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:27.027015 1113 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:27.027415 503 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:27.027485 1117 consensus_queue.cc:1035] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:27.028013 1102 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:27.028131 1113 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:27.028296 1102 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:27.039469 1150 mvcc.cc:205] Tried to move back new op lower bound from 7023152197718732800 to 7023152196463087616. Current Snapshot: MvccSnapshot[applied={T|T < 7023152197718732800}]
I20240502 07:59:27.046690 1155 mvcc.cc:205] Tried to move back new op lower bound from 7023152197725790208 to 7023152196712906752. Current Snapshot: MvccSnapshot[applied={T|T < 7023152197725790208}]
I20240502 07:59:27.060833 1149 mvcc.cc:205] Tried to move back new op lower bound from 7023152197744168960 to 7023152196138205184. Current Snapshot: MvccSnapshot[applied={T|T < 7023152197725118464}]
INSERT report
rows total: 2000
time total: 128.561 ms
time per row: 0.0642807 ms
I20240502 07:59:27.104562 32693 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46856:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 07:59:27.105160 32693 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 07:59:27.111908 350 tablet_service.cc:1467] Processing CreateTablet for tablet a921e3823fa34be9a26fc4f46e12ef28 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a353f7ca52eb4a0e86f45854ebf5dbe6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.112030 481 tablet_service.cc:1467] Processing CreateTablet for tablet a921e3823fa34be9a26fc4f46e12ef28 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a353f7ca52eb4a0e86f45854ebf5dbe6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.112264 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a921e3823fa34be9a26fc4f46e12ef28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.112334 481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a921e3823fa34be9a26fc4f46e12ef28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.112469 349 tablet_service.cc:1467] Processing CreateTablet for tablet 9bf16ec8c5854b2daaccabe49d10ba26 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a353f7ca52eb4a0e86f45854ebf5dbe6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.112689 349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bf16ec8c5854b2daaccabe49d10ba26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.114377 1178 tablet_bootstrap.cc:492] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd: Bootstrap starting.
I20240502 07:59:27.114609 1179 tablet_bootstrap.cc:492] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a: Bootstrap starting.
I20240502 07:59:27.114773 614 tablet_service.cc:1467] Processing CreateTablet for tablet a921e3823fa34be9a26fc4f46e12ef28 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a353f7ca52eb4a0e86f45854ebf5dbe6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.115105 614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a921e3823fa34be9a26fc4f46e12ef28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.115222 1178 tablet_bootstrap.cc:654] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.115538 1179 tablet_bootstrap.cc:654] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.116557 1179 tablet_bootstrap.cc:492] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a: No bootstrap required, opened a new log
I20240502 07:59:27.116649 1179 ts_tablet_manager.cc:1367] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:27.117100 1179 raft_consensus.cc:348] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.117214 1179 raft_consensus.cc:374] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.117249 1179 raft_consensus.cc:729] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a60a9fab026458293b88d08fb10e88a, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.117416 1179 consensus_queue.cc:260] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.117772 1179 ts_tablet_manager.cc:1398] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:27.118633 1182 tablet_bootstrap.cc:492] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec: Bootstrap starting.
I20240502 07:59:27.119534 1182 tablet_bootstrap.cc:654] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.120118 613 tablet_service.cc:1467] Processing CreateTablet for tablet 9bf16ec8c5854b2daaccabe49d10ba26 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a353f7ca52eb4a0e86f45854ebf5dbe6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.120369 613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bf16ec8c5854b2daaccabe49d10ba26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.120731 1182 tablet_bootstrap.cc:492] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec: No bootstrap required, opened a new log
I20240502 07:59:27.120801 1182 ts_tablet_manager.cc:1367] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:27.121209 1182 raft_consensus.cc:348] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.121320 1182 raft_consensus.cc:374] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.121363 1182 raft_consensus.cc:729] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733b523e2f3e43539b480b034726d7ec, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.121496 1182 consensus_queue.cc:260] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.121773 1182 ts_tablet_manager.cc:1398] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:27.122361 1178 tablet_bootstrap.cc:492] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd: No bootstrap required, opened a new log
I20240502 07:59:27.122463 1178 ts_tablet_manager.cc:1367] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 07:59:27.122951 1178 raft_consensus.cc:348] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.123059 1178 raft_consensus.cc:374] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.123095 1178 raft_consensus.cc:729] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca73afa63a04c378a0948e9691601fd, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.123152 349 tablet_service.cc:1467] Processing CreateTablet for tablet 71841e55040a499bac87b1db55662200 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a353f7ca52eb4a0e86f45854ebf5dbe6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.123221 1178 consensus_queue.cc:260] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.123443 349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71841e55040a499bac87b1db55662200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.123515 1178 ts_tablet_manager.cc:1398] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:27.123695 1178 tablet_bootstrap.cc:492] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd: Bootstrap starting.
I20240502 07:59:27.123725 481 tablet_service.cc:1467] Processing CreateTablet for tablet 9bf16ec8c5854b2daaccabe49d10ba26 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a353f7ca52eb4a0e86f45854ebf5dbe6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.124003 481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bf16ec8c5854b2daaccabe49d10ba26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.124873 1178 tablet_bootstrap.cc:654] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.126642 483 tablet_service.cc:1467] Processing CreateTablet for tablet 71841e55040a499bac87b1db55662200 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a353f7ca52eb4a0e86f45854ebf5dbe6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.127171 483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71841e55040a499bac87b1db55662200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.127662 613 tablet_service.cc:1467] Processing CreateTablet for tablet 71841e55040a499bac87b1db55662200 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a353f7ca52eb4a0e86f45854ebf5dbe6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.128269 613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71841e55040a499bac87b1db55662200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.131836 1179 tablet_bootstrap.cc:492] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a: Bootstrap starting.
I20240502 07:59:27.132840 1179 tablet_bootstrap.cc:654] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.133276 1182 tablet_bootstrap.cc:492] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec: Bootstrap starting.
I20240502 07:59:27.133962 1179 tablet_bootstrap.cc:492] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a: No bootstrap required, opened a new log
I20240502 07:59:27.134038 1179 ts_tablet_manager.cc:1367] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:27.134204 1182 tablet_bootstrap.cc:654] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.134531 1179 raft_consensus.cc:348] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.134634 1179 raft_consensus.cc:374] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.134673 1179 raft_consensus.cc:729] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a60a9fab026458293b88d08fb10e88a, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.134817 1179 consensus_queue.cc:260] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.135107 1178 tablet_bootstrap.cc:492] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd: No bootstrap required, opened a new log
I20240502 07:59:27.135116 1179 ts_tablet_manager.cc:1398] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:27.135210 1178 ts_tablet_manager.cc:1367] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 07:59:27.135279 1179 tablet_bootstrap.cc:492] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a: Bootstrap starting.
I20240502 07:59:27.136174 1179 tablet_bootstrap.cc:654] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.136533 1182 tablet_bootstrap.cc:492] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec: No bootstrap required, opened a new log
I20240502 07:59:27.135713 1178 raft_consensus.cc:348] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.136631 1182 ts_tablet_manager.cc:1367] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:27.136651 1178 raft_consensus.cc:374] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.136709 1178 raft_consensus.cc:729] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca73afa63a04c378a0948e9691601fd, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.136926 1178 consensus_queue.cc:260] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.137138 1182 raft_consensus.cc:348] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.137259 1182 raft_consensus.cc:374] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.137302 1182 raft_consensus.cc:729] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733b523e2f3e43539b480b034726d7ec, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.137434 1182 consensus_queue.cc:260] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.137712 1179 tablet_bootstrap.cc:492] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a: No bootstrap required, opened a new log
I20240502 07:59:27.137712 1182 ts_tablet_manager.cc:1398] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:27.137795 1179 ts_tablet_manager.cc:1367] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:27.137884 1182 tablet_bootstrap.cc:492] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec: Bootstrap starting.
I20240502 07:59:27.138211 1179 raft_consensus.cc:348] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.138320 1179 raft_consensus.cc:374] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.138352 1179 raft_consensus.cc:729] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a60a9fab026458293b88d08fb10e88a, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.138465 1179 consensus_queue.cc:260] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.138725 1179 ts_tablet_manager.cc:1398] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:27.138787 1182 tablet_bootstrap.cc:654] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.138870 1178 ts_tablet_manager.cc:1398] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:27.139071 1178 tablet_bootstrap.cc:492] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd: Bootstrap starting.
I20240502 07:59:27.140858 1182 tablet_bootstrap.cc:492] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec: No bootstrap required, opened a new log
I20240502 07:59:27.140941 1182 ts_tablet_manager.cc:1367] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:59:27.141400 1182 raft_consensus.cc:348] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.141489 1182 raft_consensus.cc:374] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.141525 1182 raft_consensus.cc:729] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733b523e2f3e43539b480b034726d7ec, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.141641 1182 consensus_queue.cc:260] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.141952 1182 ts_tablet_manager.cc:1398] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:27.144148 1178 tablet_bootstrap.cc:654] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.145458 1178 tablet_bootstrap.cc:492] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd: No bootstrap required, opened a new log
I20240502 07:59:27.145576 1178 ts_tablet_manager.cc:1367] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 07:59:27.146029 1178 raft_consensus.cc:348] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.146147 1178 raft_consensus.cc:374] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.146188 1178 raft_consensus.cc:729] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca73afa63a04c378a0948e9691601fd, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.146348 1178 consensus_queue.cc:260] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.146665 1178 ts_tablet_manager.cc:1398] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:27.165953 1170 raft_consensus.cc:482] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:27.166105 1170 raft_consensus.cc:504] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.166524 1170 leader_election.cc:288] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429), 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457)
I20240502 07:59:27.166952 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf16ec8c5854b2daaccabe49d10ba26" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca73afa63a04c378a0948e9691601fd" is_pre_election: true
I20240502 07:59:27.167117 370 raft_consensus.cc:2454] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 0.
I20240502 07:59:27.167066 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf16ec8c5854b2daaccabe49d10ba26" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a" is_pre_election: true
I20240502 07:59:27.167222 501 raft_consensus.cc:2454] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 0.
I20240502 07:59:27.167549 568 leader_election.cc:302] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca73afa63a04c378a0948e9691601fd, 733b523e2f3e43539b480b034726d7ec; no voters:
I20240502 07:59:27.167722 1170 raft_consensus.cc:2781] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:27.167878 1170 raft_consensus.cc:482] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:27.167927 1170 raft_consensus.cc:3037] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.168787 1170 raft_consensus.cc:504] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.169128 1170 leader_election.cc:288] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 election: Requested vote from peers 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429), 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457)
I20240502 07:59:27.169482 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf16ec8c5854b2daaccabe49d10ba26" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca73afa63a04c378a0948e9691601fd"
I20240502 07:59:27.169612 370 raft_consensus.cc:3037] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.169627 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bf16ec8c5854b2daaccabe49d10ba26" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a"
I20240502 07:59:27.169741 501 raft_consensus.cc:3037] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.170747 501 raft_consensus.cc:2454] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 1.
I20240502 07:59:27.170746 370 raft_consensus.cc:2454] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 1.
I20240502 07:59:27.171059 568 leader_election.cc:302] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a60a9fab026458293b88d08fb10e88a, 733b523e2f3e43539b480b034726d7ec; no voters:
I20240502 07:59:27.171248 1170 raft_consensus.cc:2781] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:27.171339 1170 raft_consensus.cc:686] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Becoming Leader. State: Replica: 733b523e2f3e43539b480b034726d7ec, State: Running, Role: LEADER
I20240502 07:59:27.171479 1170 consensus_queue.cc:237] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.174013 32695 catalog_manager.cc:5484] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 733b523e2f3e43539b480b034726d7ec (127.18.13.131). New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:27.251435 1170 raft_consensus.cc:482] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:27.251564 1170 raft_consensus.cc:504] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.252020 1170 leader_election.cc:288] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429), 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457)
I20240502 07:59:27.252466 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71841e55040a499bac87b1db55662200" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca73afa63a04c378a0948e9691601fd" is_pre_election: true
I20240502 07:59:27.252583 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71841e55040a499bac87b1db55662200" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a" is_pre_election: true
I20240502 07:59:27.252648 370 raft_consensus.cc:2454] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 0.
I20240502 07:59:27.252712 501 raft_consensus.cc:2454] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 0.
I20240502 07:59:27.253017 568 leader_election.cc:302] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca73afa63a04c378a0948e9691601fd, 733b523e2f3e43539b480b034726d7ec; no voters:
I20240502 07:59:27.253221 1170 raft_consensus.cc:2781] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:27.253291 1170 raft_consensus.cc:482] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:27.253340 1170 raft_consensus.cc:3037] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.254226 1170 raft_consensus.cc:504] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.254602 1170 leader_election.cc:288] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 election: Requested vote from peers 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429), 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457)
I20240502 07:59:27.254963 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71841e55040a499bac87b1db55662200" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca73afa63a04c378a0948e9691601fd"
I20240502 07:59:27.255076 370 raft_consensus.cc:3037] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.255146 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71841e55040a499bac87b1db55662200" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a"
I20240502 07:59:27.255263 501 raft_consensus.cc:3037] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.256129 370 raft_consensus.cc:2454] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 1.
I20240502 07:59:27.256129 501 raft_consensus.cc:2454] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 1.
I20240502 07:59:27.256470 568 leader_election.cc:302] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a60a9fab026458293b88d08fb10e88a, 733b523e2f3e43539b480b034726d7ec; no voters:
I20240502 07:59:27.256714 1170 raft_consensus.cc:2781] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:27.256798 1170 raft_consensus.cc:686] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Becoming Leader. State: Replica: 733b523e2f3e43539b480b034726d7ec, State: Running, Role: LEADER
I20240502 07:59:27.256951 1170 consensus_queue.cc:237] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.258719 32695 catalog_manager.cc:5484] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 733b523e2f3e43539b480b034726d7ec (127.18.13.131). New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:27.362978 1107 raft_consensus.cc:482] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:27.363102 1107 raft_consensus.cc:504] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.363543 1107 leader_election.cc:288] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457), 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:27.363981 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a921e3823fa34be9a26fc4f46e12ef28" candidate_uuid: "2ca73afa63a04c378a0948e9691601fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a" is_pre_election: true
I20240502 07:59:27.363991 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a921e3823fa34be9a26fc4f46e12ef28" candidate_uuid: "2ca73afa63a04c378a0948e9691601fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733b523e2f3e43539b480b034726d7ec" is_pre_election: true
I20240502 07:59:27.364130 501 raft_consensus.cc:2454] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca73afa63a04c378a0948e9691601fd in term 0.
I20240502 07:59:27.364133 635 raft_consensus.cc:2454] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca73afa63a04c378a0948e9691601fd in term 0.
I20240502 07:59:27.364416 306 leader_election.cc:302] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca73afa63a04c378a0948e9691601fd, 733b523e2f3e43539b480b034726d7ec; no voters:
I20240502 07:59:27.364614 1107 raft_consensus.cc:2781] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:27.364665 1107 raft_consensus.cc:482] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:27.364710 1107 raft_consensus.cc:3037] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.365526 1107 raft_consensus.cc:504] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.365851 1107 leader_election.cc:288] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [CANDIDATE]: Term 1 election: Requested vote from peers 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457), 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:27.366189 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a921e3823fa34be9a26fc4f46e12ef28" candidate_uuid: "2ca73afa63a04c378a0948e9691601fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733b523e2f3e43539b480b034726d7ec"
I20240502 07:59:27.366310 635 raft_consensus.cc:3037] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.367246 635 raft_consensus.cc:2454] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca73afa63a04c378a0948e9691601fd in term 1.
I20240502 07:59:27.367458 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a921e3823fa34be9a26fc4f46e12ef28" candidate_uuid: "2ca73afa63a04c378a0948e9691601fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a"
I20240502 07:59:27.367555 306 leader_election.cc:302] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca73afa63a04c378a0948e9691601fd, 733b523e2f3e43539b480b034726d7ec; no voters:
I20240502 07:59:27.367591 501 raft_consensus.cc:3037] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.367719 1107 raft_consensus.cc:2781] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:27.367823 1107 raft_consensus.cc:686] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEADER]: Becoming Leader. State: Replica: 2ca73afa63a04c378a0948e9691601fd, State: Running, Role: LEADER
I20240502 07:59:27.367964 1107 consensus_queue.cc:237] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.368495 501 raft_consensus.cc:2454] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca73afa63a04c378a0948e9691601fd in term 1.
I20240502 07:59:27.369681 32695 catalog_manager.cc:5484] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ca73afa63a04c378a0948e9691601fd (127.18.13.129). New cstate: current_term: 1 leader_uuid: "2ca73afa63a04c378a0948e9691601fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:27.526417 1193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:27.526708 1193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:27.529814 1193 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 07:59:27.606689 501 raft_consensus.cc:1261] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:27.606689 502 raft_consensus.cc:1261] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:27.607739 502 raft_consensus.cc:1261] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:27.608129 635 raft_consensus.cc:1261] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:27.608476 1117 consensus_queue.cc:1035] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:27.608781 1117 consensus_queue.cc:1035] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:27.609213 370 raft_consensus.cc:1261] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:27.609186 1170 consensus_queue.cc:1035] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:27.609221 1102 consensus_queue.cc:1035] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:27.610879 1170 consensus_queue.cc:1035] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:27.613831 371 raft_consensus.cc:1261] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:27.615835 1170 consensus_queue.cc:1035] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.55 ms
time per row: 0.0522752 ms
I20240502 07:59:27.676357 32694 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46856:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 07:59:27.676913 32694 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 07:59:27.682296 349 tablet_service.cc:1467] Processing CreateTablet for tablet 590180286acf49189ba03c8725946565 (DEFAULT_TABLE table=rebalance_test_table_2 [id=63eb7086af244504b9daf9beaeb2a9bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.682292 483 tablet_service.cc:1467] Processing CreateTablet for tablet 590180286acf49189ba03c8725946565 (DEFAULT_TABLE table=rebalance_test_table_2 [id=63eb7086af244504b9daf9beaeb2a9bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.682629 349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590180286acf49189ba03c8725946565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.682629 483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590180286acf49189ba03c8725946565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.682998 614 tablet_service.cc:1467] Processing CreateTablet for tablet d0d4cda1deb247d1872658e08dbd6d6b (DEFAULT_TABLE table=rebalance_test_table_2 [id=63eb7086af244504b9daf9beaeb2a9bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.683223 614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d4cda1deb247d1872658e08dbd6d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.684139 481 tablet_service.cc:1467] Processing CreateTablet for tablet d0d4cda1deb247d1872658e08dbd6d6b (DEFAULT_TABLE table=rebalance_test_table_2 [id=63eb7086af244504b9daf9beaeb2a9bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.684387 481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d4cda1deb247d1872658e08dbd6d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.685122 613 tablet_service.cc:1467] Processing CreateTablet for tablet 590180286acf49189ba03c8725946565 (DEFAULT_TABLE table=rebalance_test_table_2 [id=63eb7086af244504b9daf9beaeb2a9bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.685307 613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590180286acf49189ba03c8725946565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.685474 1253 tablet_bootstrap.cc:492] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec: Bootstrap starting.
I20240502 07:59:27.686333 1253 tablet_bootstrap.cc:654] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.686564 614 tablet_service.cc:1467] Processing CreateTablet for tablet e199f4ff646b4992ab7c6d5527ac32db (DEFAULT_TABLE table=rebalance_test_table_2 [id=63eb7086af244504b9daf9beaeb2a9bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.686782 614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e199f4ff646b4992ab7c6d5527ac32db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.686821 483 tablet_service.cc:1467] Processing CreateTablet for tablet e199f4ff646b4992ab7c6d5527ac32db (DEFAULT_TABLE table=rebalance_test_table_2 [id=63eb7086af244504b9daf9beaeb2a9bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.687031 483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e199f4ff646b4992ab7c6d5527ac32db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.688071 350 tablet_service.cc:1467] Processing CreateTablet for tablet e199f4ff646b4992ab7c6d5527ac32db (DEFAULT_TABLE table=rebalance_test_table_2 [id=63eb7086af244504b9daf9beaeb2a9bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.688377 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e199f4ff646b4992ab7c6d5527ac32db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.688722 349 tablet_service.cc:1467] Processing CreateTablet for tablet d0d4cda1deb247d1872658e08dbd6d6b (DEFAULT_TABLE table=rebalance_test_table_2 [id=63eb7086af244504b9daf9beaeb2a9bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:27.688930 349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d4cda1deb247d1872658e08dbd6d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:27.689289 1251 tablet_bootstrap.cc:492] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd: Bootstrap starting.
I20240502 07:59:27.690055 1251 tablet_bootstrap.cc:654] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.690615 1252 tablet_bootstrap.cc:492] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a: Bootstrap starting.
I20240502 07:59:27.691035 1251 tablet_bootstrap.cc:492] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd: No bootstrap required, opened a new log
I20240502 07:59:27.691138 1251 ts_tablet_manager.cc:1367] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:27.691300 1253 tablet_bootstrap.cc:492] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec: No bootstrap required, opened a new log
I20240502 07:59:27.691488 1253 ts_tablet_manager.cc:1367] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:59:27.691593 1252 tablet_bootstrap.cc:654] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.692014 1251 raft_consensus.cc:348] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.692073 1253 raft_consensus.cc:348] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.692142 1251 raft_consensus.cc:374] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.692193 1251 raft_consensus.cc:729] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca73afa63a04c378a0948e9691601fd, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.692209 1253 raft_consensus.cc:374] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.692251 1253 raft_consensus.cc:729] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733b523e2f3e43539b480b034726d7ec, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.692405 1251 consensus_queue.cc:260] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.692409 1253 consensus_queue.cc:260] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.692698 1251 ts_tablet_manager.cc:1398] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:27.692751 1252 tablet_bootstrap.cc:492] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a: No bootstrap required, opened a new log
I20240502 07:59:27.692783 1253 ts_tablet_manager.cc:1398] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:27.692842 1252 ts_tablet_manager.cc:1367] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 07:59:27.692859 1251 tablet_bootstrap.cc:492] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd: Bootstrap starting.
I20240502 07:59:27.692999 1253 tablet_bootstrap.cc:492] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec: Bootstrap starting.
I20240502 07:59:27.693259 1252 raft_consensus.cc:348] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.693363 1252 raft_consensus.cc:374] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.693397 1252 raft_consensus.cc:729] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a60a9fab026458293b88d08fb10e88a, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.693528 1252 consensus_queue.cc:260] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.693825 1252 ts_tablet_manager.cc:1398] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:27.693871 1253 tablet_bootstrap.cc:654] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.693991 1252 tablet_bootstrap.cc:492] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a: Bootstrap starting.
I20240502 07:59:27.694835 1251 tablet_bootstrap.cc:654] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.694918 1252 tablet_bootstrap.cc:654] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.696060 1252 tablet_bootstrap.cc:492] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a: No bootstrap required, opened a new log
I20240502 07:59:27.696147 1252 ts_tablet_manager.cc:1367] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:59:27.696615 1252 raft_consensus.cc:348] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.696759 1252 raft_consensus.cc:374] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.696815 1252 raft_consensus.cc:729] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a60a9fab026458293b88d08fb10e88a, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.696978 1252 consensus_queue.cc:260] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.697269 1252 ts_tablet_manager.cc:1398] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:27.697475 1252 tablet_bootstrap.cc:492] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: Bootstrap starting.
I20240502 07:59:27.698076 1253 tablet_bootstrap.cc:492] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec: No bootstrap required, opened a new log
I20240502 07:59:27.698186 1253 ts_tablet_manager.cc:1367] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240502 07:59:27.698711 1252 tablet_bootstrap.cc:654] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.698663 1253 raft_consensus.cc:348] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.698779 1253 raft_consensus.cc:374] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.698824 1253 raft_consensus.cc:729] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733b523e2f3e43539b480b034726d7ec, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.698966 1253 consensus_queue.cc:260] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.699296 1253 ts_tablet_manager.cc:1398] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:27.699553 1253 tablet_bootstrap.cc:492] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec: Bootstrap starting.
I20240502 07:59:27.700001 1252 tablet_bootstrap.cc:492] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: No bootstrap required, opened a new log
I20240502 07:59:27.700022 1251 tablet_bootstrap.cc:492] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd: No bootstrap required, opened a new log
I20240502 07:59:27.700084 1252 ts_tablet_manager.cc:1367] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:27.700227 1251 ts_tablet_manager.cc:1367] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 07:59:27.700516 1252 raft_consensus.cc:348] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.700635 1252 raft_consensus.cc:374] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.700678 1252 raft_consensus.cc:729] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a60a9fab026458293b88d08fb10e88a, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.700835 1252 consensus_queue.cc:260] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.700852 1251 raft_consensus.cc:348] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.701108 1252 ts_tablet_manager.cc:1398] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:27.701473 1253 tablet_bootstrap.cc:654] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.701032 1251 raft_consensus.cc:374] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.701665 1251 raft_consensus.cc:729] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca73afa63a04c378a0948e9691601fd, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.701835 1251 consensus_queue.cc:260] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.702122 1251 ts_tablet_manager.cc:1398] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:27.702334 1251 tablet_bootstrap.cc:492] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd: Bootstrap starting.
I20240502 07:59:27.702831 1253 tablet_bootstrap.cc:492] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec: No bootstrap required, opened a new log
I20240502 07:59:27.702910 1253 ts_tablet_manager.cc:1367] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:27.703362 1253 raft_consensus.cc:348] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.703465 1253 raft_consensus.cc:374] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.703497 1253 raft_consensus.cc:729] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 733b523e2f3e43539b480b034726d7ec, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.703616 1253 consensus_queue.cc:260] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.703920 1253 ts_tablet_manager.cc:1398] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:27.704162 1251 tablet_bootstrap.cc:654] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:27.705313 1251 tablet_bootstrap.cc:492] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd: No bootstrap required, opened a new log
I20240502 07:59:27.705410 1251 ts_tablet_manager.cc:1367] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:27.705842 1251 raft_consensus.cc:348] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.705929 1251 raft_consensus.cc:374] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:27.705950 1251 raft_consensus.cc:729] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca73afa63a04c378a0948e9691601fd, State: Initialized, Role: FOLLOWER
I20240502 07:59:27.706048 1251 consensus_queue.cc:260] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.706226 1251 ts_tablet_manager.cc:1398] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:27.709105 1245 raft_consensus.cc:482] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:27.709216 1245 raft_consensus.cc:504] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.709718 1245 leader_election.cc:288] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429), 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457)
I20240502 07:59:27.709914 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e199f4ff646b4992ab7c6d5527ac32db" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca73afa63a04c378a0948e9691601fd" is_pre_election: true
I20240502 07:59:27.710016 371 raft_consensus.cc:2454] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 0.
I20240502 07:59:27.710045 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e199f4ff646b4992ab7c6d5527ac32db" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a" is_pre_election: true
I20240502 07:59:27.710186 501 raft_consensus.cc:2454] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 0.
I20240502 07:59:27.710309 568 leader_election.cc:302] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca73afa63a04c378a0948e9691601fd, 733b523e2f3e43539b480b034726d7ec; no voters:
I20240502 07:59:27.710479 1245 raft_consensus.cc:2781] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:27.710547 1245 raft_consensus.cc:482] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:27.710597 1245 raft_consensus.cc:3037] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.711393 1245 raft_consensus.cc:504] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.711683 1245 leader_election.cc:288] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 election: Requested vote from peers 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429), 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457)
I20240502 07:59:27.711985 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e199f4ff646b4992ab7c6d5527ac32db" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca73afa63a04c378a0948e9691601fd"
I20240502 07:59:27.712077 371 raft_consensus.cc:3037] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.712045 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e199f4ff646b4992ab7c6d5527ac32db" candidate_uuid: "733b523e2f3e43539b480b034726d7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a"
I20240502 07:59:27.712164 501 raft_consensus.cc:3037] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.712777 371 raft_consensus.cc:2454] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 1.
I20240502 07:59:27.712880 501 raft_consensus.cc:2454] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 733b523e2f3e43539b480b034726d7ec in term 1.
I20240502 07:59:27.713040 568 leader_election.cc:302] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca73afa63a04c378a0948e9691601fd, 733b523e2f3e43539b480b034726d7ec; no voters:
I20240502 07:59:27.713186 1245 raft_consensus.cc:2781] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:27.713270 1245 raft_consensus.cc:686] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Becoming Leader. State: Replica: 733b523e2f3e43539b480b034726d7ec, State: Running, Role: LEADER
I20240502 07:59:27.713400 1245 consensus_queue.cc:237] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.715099 32694 catalog_manager.cc:5484] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 733b523e2f3e43539b480b034726d7ec (127.18.13.131). New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:27.798481 1233 raft_consensus.cc:482] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:27.798626 1233 raft_consensus.cc:504] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.799708 1233 leader_election.cc:288] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429), 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:27.804399 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d4cda1deb247d1872658e08dbd6d6b" candidate_uuid: "5a60a9fab026458293b88d08fb10e88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733b523e2f3e43539b480b034726d7ec" is_pre_election: true
I20240502 07:59:27.804394 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d4cda1deb247d1872658e08dbd6d6b" candidate_uuid: "5a60a9fab026458293b88d08fb10e88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca73afa63a04c378a0948e9691601fd" is_pre_election: true
I20240502 07:59:27.804540 635 raft_consensus.cc:2454] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a60a9fab026458293b88d08fb10e88a in term 0.
I20240502 07:59:27.804548 371 raft_consensus.cc:2454] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a60a9fab026458293b88d08fb10e88a in term 0.
I20240502 07:59:27.804895 438 leader_election.cc:302] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a60a9fab026458293b88d08fb10e88a, 733b523e2f3e43539b480b034726d7ec; no voters:
I20240502 07:59:27.805145 1233 raft_consensus.cc:2781] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:27.805204 1233 raft_consensus.cc:482] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:27.805233 1233 raft_consensus.cc:3037] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.806078 1233 raft_consensus.cc:504] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.806439 1233 leader_election.cc:288] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [CANDIDATE]: Term 1 election: Requested vote from peers 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429), 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:27.806757 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d4cda1deb247d1872658e08dbd6d6b" candidate_uuid: "5a60a9fab026458293b88d08fb10e88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca73afa63a04c378a0948e9691601fd"
I20240502 07:59:27.806815 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d4cda1deb247d1872658e08dbd6d6b" candidate_uuid: "5a60a9fab026458293b88d08fb10e88a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733b523e2f3e43539b480b034726d7ec"
I20240502 07:59:27.806859 371 raft_consensus.cc:3037] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.806943 635 raft_consensus.cc:3037] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.807570 371 raft_consensus.cc:2454] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a60a9fab026458293b88d08fb10e88a in term 1.
I20240502 07:59:27.807682 635 raft_consensus.cc:2454] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a60a9fab026458293b88d08fb10e88a in term 1.
I20240502 07:59:27.807878 436 leader_election.cc:302] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca73afa63a04c378a0948e9691601fd, 5a60a9fab026458293b88d08fb10e88a; no voters:
I20240502 07:59:27.808018 1233 raft_consensus.cc:2781] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:27.808130 1233 raft_consensus.cc:686] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [term 1 LEADER]: Becoming Leader. State: Replica: 5a60a9fab026458293b88d08fb10e88a, State: Running, Role: LEADER
I20240502 07:59:27.808416 1233 consensus_queue.cc:237] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.810688 32694 catalog_manager.cc:5484] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a60a9fab026458293b88d08fb10e88a (127.18.13.130). New cstate: current_term: 1 leader_uuid: "5a60a9fab026458293b88d08fb10e88a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:27.814482 1107 raft_consensus.cc:482] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:27.814574 1107 raft_consensus.cc:504] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.814910 1107 leader_election.cc:288] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457), 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:27.815243 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590180286acf49189ba03c8725946565" candidate_uuid: "2ca73afa63a04c378a0948e9691601fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a" is_pre_election: true
I20240502 07:59:27.815279 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590180286acf49189ba03c8725946565" candidate_uuid: "2ca73afa63a04c378a0948e9691601fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733b523e2f3e43539b480b034726d7ec" is_pre_election: true
I20240502 07:59:27.815366 501 raft_consensus.cc:2454] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca73afa63a04c378a0948e9691601fd in term 0.
I20240502 07:59:27.815395 635 raft_consensus.cc:2454] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca73afa63a04c378a0948e9691601fd in term 0.
I20240502 07:59:27.815640 304 leader_election.cc:302] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca73afa63a04c378a0948e9691601fd, 5a60a9fab026458293b88d08fb10e88a; no voters:
I20240502 07:59:27.815824 1107 raft_consensus.cc:2781] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:27.815882 1107 raft_consensus.cc:482] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:27.815932 1107 raft_consensus.cc:3037] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.816633 1107 raft_consensus.cc:504] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.816957 1107 leader_election.cc:288] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [CANDIDATE]: Term 1 election: Requested vote from peers 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457), 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:27.817247 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590180286acf49189ba03c8725946565" candidate_uuid: "2ca73afa63a04c378a0948e9691601fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a60a9fab026458293b88d08fb10e88a"
I20240502 07:59:27.817307 635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590180286acf49189ba03c8725946565" candidate_uuid: "2ca73afa63a04c378a0948e9691601fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "733b523e2f3e43539b480b034726d7ec"
I20240502 07:59:27.817365 501 raft_consensus.cc:3037] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.817416 635 raft_consensus.cc:3037] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:27.818058 501 raft_consensus.cc:2454] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca73afa63a04c378a0948e9691601fd in term 1.
I20240502 07:59:27.818135 635 raft_consensus.cc:2454] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca73afa63a04c378a0948e9691601fd in term 1.
I20240502 07:59:27.818356 304 leader_election.cc:302] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca73afa63a04c378a0948e9691601fd, 5a60a9fab026458293b88d08fb10e88a; no voters:
I20240502 07:59:27.818512 1107 raft_consensus.cc:2781] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:27.818603 1107 raft_consensus.cc:686] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEADER]: Becoming Leader. State: Replica: 2ca73afa63a04c378a0948e9691601fd, State: Running, Role: LEADER
I20240502 07:59:27.818734 1107 consensus_queue.cc:237] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } }
I20240502 07:59:27.820308 32694 catalog_manager.cc:5484] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ca73afa63a04c378a0948e9691601fd (127.18.13.129). New cstate: current_term: 1 leader_uuid: "2ca73afa63a04c378a0948e9691601fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:27.952001 1268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:27.952277 1268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:27.955437 1268 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 07:59:28.046809 501 raft_consensus.cc:1261] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:28.047885 635 raft_consensus.cc:1261] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Refusing update from remote peer 5a60a9fab026458293b88d08fb10e88a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:28.048084 634 raft_consensus.cc:1261] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:28.048393 1229 consensus_queue.cc:1035] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:28.048624 1107 consensus_queue.cc:1035] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:28.053977 371 raft_consensus.cc:1261] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 5a60a9fab026458293b88d08fb10e88a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:28.054057 502 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:28.054059 370 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:28.054427 1107 consensus_queue.cc:1035] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:28.054807 1102 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:28.055023 1245 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:28.055429 1233 consensus_queue.cc:1035] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 112.843 ms
time per row: 0.0564214 ms
I20240502 07:59:28.114970 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.132:33885
--local_ip_for_outbound_sockets=127.18.13.132
--tserver_master_addrs=127.18.13.190:43401
--webserver_port=44375
--webserver_interface=127.18.13.132
--builtin_ntp_servers=127.18.13.148:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 07:59:28.226935 1305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:28.227161 1305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:28.227210 1305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:28.230573 1305 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:28.230649 1305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:28.230747 1305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.132
I20240502 07:59:28.234361 1305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.132:33885
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.18.13.132
--webserver_port=44375
--tserver_master_addrs=127.18.13.190:43401
--never_fsync=true
--heap_profile_path=/tmp/kudu.1305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.132
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:28.235270 1305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:28.236430 1305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:28.244336 1310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:28.244959 1311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:28.245640 1313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:28.246341 1305 server_base.cc:1034] running on GCE node
I20240502 07:59:28.246742 1305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:28.247642 1305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:28.248800 1305 hybrid_clock.cc:648] HybridClock initialized: now 1714636768248783 us; error 48 us; skew 500 ppm
I20240502 07:59:28.250823 1305 webserver.cc:460] Webserver started at http://127.18.13.132:44375/ using document root <none> and password file <none>
I20240502 07:59:28.251364 1305 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:28.251431 1305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:28.254861 1305 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:28.256780 1319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:28.257768 1305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:28.257879 1305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "468fd6f1f0fb46ce87ab94f922e89661"
format_stamp: "Formatted at 2024-05-02 07:59:22 on dist-test-slave-f46z"
I20240502 07:59:28.258294 1305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:28.271768 1305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:28.272485 1305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:28.272647 1305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:28.273183 1305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:28.274046 1305 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:28.274104 1305 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:28.274163 1305 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:28.274187 1305 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:28.282941 1305 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.132:33885
I20240502 07:59:28.282965 1432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.132:33885 every 8 connection(s)
I20240502 07:59:28.284044 1305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:59:28.291000 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 1305
I20240502 07:59:28.291487 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.134:39939
--local_ip_for_outbound_sockets=127.18.13.134
--tserver_master_addrs=127.18.13.190:43401
--webserver_port=41527
--webserver_interface=127.18.13.134
--builtin_ntp_servers=127.18.13.148:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:28.293205 1433 heartbeater.cc:347] Connected to a master server at 127.18.13.190:43401
I20240502 07:59:28.293442 1433 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:28.294003 1433 heartbeater.cc:511] Master 127.18.13.190:43401 requested a full tablet report, sending...
I20240502 07:59:28.295408 32696 ts_manager.cc:194] Re-registered known tserver with Master: 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132:33885)
I20240502 07:59:28.296322 32696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.132:49265
W20240502 07:59:28.401976 1436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:28.402215 1436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:28.402261 1436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:28.405373 1436 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:28.405444 1436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:28.405515 1436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.134
I20240502 07:59:28.409209 1436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.134:39939
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.18.13.134
--webserver_port=41527
--tserver_master_addrs=127.18.13.190:43401
--never_fsync=true
--heap_profile_path=/tmp/kudu.1436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.134
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:28.410148 1436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:28.411278 1436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:28.420447 1444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:28.420552 1441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:28.421221 1442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:28.421931 1436 server_base.cc:1034] running on GCE node
I20240502 07:59:28.422331 1436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:28.423007 1436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:28.424165 1436 hybrid_clock.cc:648] HybridClock initialized: now 1714636768424127 us; error 49 us; skew 500 ppm
I20240502 07:59:28.426108 1436 webserver.cc:460] Webserver started at http://127.18.13.134:41527/ using document root <none> and password file <none>
I20240502 07:59:28.426645 1436 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:28.426712 1436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:28.430146 1436 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:28.432348 1450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:28.433391 1436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:28.433485 1436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9"
format_stamp: "Formatted at 2024-05-02 07:59:22 on dist-test-slave-f46z"
I20240502 07:59:28.433904 1436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:28.447695 1436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:28.448315 1436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:28.448453 1436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:28.448976 1436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:28.449833 1436 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:28.449878 1436 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:28.449918 1436 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:28.449940 1436 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:28.459051 1436 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.134:39939
I20240502 07:59:28.459074 1563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.134:39939 every 8 connection(s)
I20240502 07:59:28.460227 1436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 07:59:28.467912 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 1436
I20240502 07:59:28.468408 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.133:44143
--local_ip_for_outbound_sockets=127.18.13.133
--tserver_master_addrs=127.18.13.190:43401
--webserver_port=33513
--webserver_interface=127.18.13.133
--builtin_ntp_servers=127.18.13.148:42219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:28.469317 1564 heartbeater.cc:347] Connected to a master server at 127.18.13.190:43401
I20240502 07:59:28.469588 1564 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:28.470259 1564 heartbeater.cc:511] Master 127.18.13.190:43401 requested a full tablet report, sending...
I20240502 07:59:28.471333 32696 ts_manager.cc:194] Re-registered known tserver with Master: 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134:39939)
I20240502 07:59:28.471963 32696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.134:43641
W20240502 07:59:28.584461 1567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:28.584724 1567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:28.584807 1567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:28.587985 1567 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:28.588068 1567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:28.588162 1567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.133
I20240502 07:59:28.592216 1567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:42219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.133:44143
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.18.13.133
--webserver_port=33513
--tserver_master_addrs=127.18.13.190:43401
--never_fsync=true
--heap_profile_path=/tmp/kudu.1567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.133
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:28.593187 1567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:28.594461 1567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:28.602864 1572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:28.603397 1575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:28.604136 1573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:28.604815 1567 server_base.cc:1034] running on GCE node
I20240502 07:59:28.605221 1567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:28.605880 1567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:28.607034 1567 hybrid_clock.cc:648] HybridClock initialized: now 1714636768607007 us; error 41 us; skew 500 ppm
I20240502 07:59:28.609185 1567 webserver.cc:460] Webserver started at http://127.18.13.133:33513/ using document root <none> and password file <none>
I20240502 07:59:28.609802 1567 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:28.609886 1567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:28.613574 1567 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20240502 07:59:28.615658 1581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:28.616761 1567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 07:59:28.616896 1567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "857b0b25994641599b6e146cec879443"
format_stamp: "Formatted at 2024-05-02 07:59:22 on dist-test-slave-f46z"
I20240502 07:59:28.617336 1567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:28.633688 1567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:28.634343 1567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:28.634519 1567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:28.635066 1567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:28.635962 1567 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:28.636014 1567 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:28.636070 1567 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:28.636092 1567 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:28.644618 1567 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.133:44143
I20240502 07:59:28.644675 1694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.133:44143 every 8 connection(s)
I20240502 07:59:28.645684 1567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:59:28.654599 1695 heartbeater.cc:347] Connected to a master server at 127.18.13.190:43401
I20240502 07:59:28.654867 1695 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:28.655057 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 1567
I20240502 07:59:28.655498 1695 heartbeater.cc:511] Master 127.18.13.190:43401 requested a full tablet report, sending...
I20240502 07:59:28.656664 32696 ts_manager.cc:194] Re-registered known tserver with Master: 857b0b25994641599b6e146cec879443 (127.18.13.133:44143)
I20240502 07:59:28.657275 32696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.133:55799
I20240502 07:59:28.853370 614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:28.854238 349 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:28.855063 1498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:28.857290 1367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:28.860358 1629 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:28.883476 483 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:28.965921 614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:28.967077 1498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:28.974283 349 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:28.982018 1367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:28.982681 483 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:28.997038 1629 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:29.102063 1498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:29.102102 614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:29.102336 1367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:29.106204 349 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:29.120725 483 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:29.129750 1629 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:29.207813 349 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:29.208104 1367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:29.210114 1498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:29.210640 1629 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:29.211388 614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:29.218506 483 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:29.258316 634 consensus_queue.cc:237] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } }
I20240502 07:59:29.260207 371 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:29.260207 501 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:29.260674 1848 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:29.260908 1102 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:29.263016 1847 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } }
I20240502 07:59:29.264322 501 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } }
I20240502 07:59:29.264322 371 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } }
W20240502 07:59:29.264990 568 consensus_peers.cc:485] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec -> Peer 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132:33885): Couldn't send request to peer 468fd6f1f0fb46ce87ab94f922e89661. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc751cb753a840d28fb8787ca1ecafb0. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:29.267076 32696 catalog_manager.cc:5484] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index -1 to 4, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:29.274322 634 consensus_queue.cc:237] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } }
I20240502 07:59:29.275893 371 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:29.275928 501 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:29.276365 1847 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:29.276544 1848 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:29.278508 1848 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } } }
I20240502 07:59:29.278914 501 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } } }
W20240502 07:59:29.279891 569 consensus_peers.cc:485] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec -> Peer 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134:39939): Couldn't send request to peer 783ed3162fe14b76b9cb7a7a6b8bdaa9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e199f4ff646b4992ab7c6d5527ac32db. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:29.280678 371 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } } }
I20240502 07:59:29.281324 32695 catalog_manager.cc:5484] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index -1 to 4, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:29.298282 371 consensus_queue.cc:237] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } }
I20240502 07:59:29.298413 1433 heartbeater.cc:503] Master 127.18.13.190:43401 was elected leader, sending a full tablet report...
I20240502 07:59:29.300170 634 raft_consensus.cc:1261] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:29.300257 501 raft_consensus.cc:1261] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:29.300592 1871 consensus_queue.cc:1035] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:29.300810 1302 consensus_queue.cc:1035] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:29.302578 1870 raft_consensus.cc:2932] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 857b0b25994641599b6e146cec879443 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } } }
I20240502 07:59:29.304020 634 raft_consensus.cc:2932] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 857b0b25994641599b6e146cec879443 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } } }
I20240502 07:59:29.305346 32696 catalog_manager.cc:5484] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd reported cstate change: config changed from index -1 to 4, NON_VOTER 857b0b25994641599b6e146cec879443 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "2ca73afa63a04c378a0948e9691601fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:29.305814 304 consensus_peers.cc:485] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd -> Peer 857b0b25994641599b6e146cec879443 (127.18.13.133:44143): Couldn't send request to peer 857b0b25994641599b6e146cec879443. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a921e3823fa34be9a26fc4f46e12ef28. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:29.306335 501 raft_consensus.cc:2932] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 857b0b25994641599b6e146cec879443 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } } }
I20240502 07:59:29.313012 634 consensus_queue.cc:237] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } }
I20240502 07:59:29.314432 371 raft_consensus.cc:1261] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:29.314589 501 raft_consensus.cc:1261] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:59:29.314584 568 consensus_peers.cc:485] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec -> Peer 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132:33885): Couldn't send request to peer 468fd6f1f0fb46ce87ab94f922e89661. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71841e55040a499bac87b1db55662200. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:29.314821 1848 consensus_queue.cc:1035] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:29.315012 1102 consensus_queue.cc:1035] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:29.316650 1102 raft_consensus.cc:2932] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } }
I20240502 07:59:29.317070 501 raft_consensus.cc:2932] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } }
I20240502 07:59:29.318203 371 raft_consensus.cc:2932] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } }
I20240502 07:59:29.319312 32692 catalog_manager.cc:5484] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index -1 to 4, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:29.326596 371 consensus_queue.cc:237] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } }
I20240502 07:59:29.328063 501 raft_consensus.cc:1261] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:29.328399 634 raft_consensus.cc:1261] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:29.328490 1302 consensus_queue.cc:1035] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:29.328814 1871 consensus_queue.cc:1035] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:29.330282 1302 raft_consensus.cc:2932] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } } }
I20240502 07:59:29.330648 501 raft_consensus.cc:2932] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } } }
W20240502 07:59:29.332121 305 consensus_peers.cc:485] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd -> Peer 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134:39939): Couldn't send request to peer 783ed3162fe14b76b9cb7a7a6b8bdaa9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 590180286acf49189ba03c8725946565. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:29.332444 634 raft_consensus.cc:2932] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } } }
I20240502 07:59:29.333091 32695 catalog_manager.cc:5484] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a reported cstate change: config changed from index -1 to 4, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "2ca73afa63a04c378a0948e9691601fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } } }
I20240502 07:59:29.340242 371 consensus_queue.cc:237] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } }
I20240502 07:59:29.341667 634 raft_consensus.cc:1261] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:29.341583 501 raft_consensus.cc:1261] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:59:29.341761 304 consensus_peers.cc:485] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd -> Peer 857b0b25994641599b6e146cec879443 (127.18.13.133:44143): Couldn't send request to peer 857b0b25994641599b6e146cec879443. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b51370efd02a42ec920c531abf4834b9. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:29.342099 1870 consensus_queue.cc:1035] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:29.342242 1871 consensus_queue.cc:1035] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:29.344012 1302 raft_consensus.cc:2932] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 857b0b25994641599b6e146cec879443 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } } }
I20240502 07:59:29.344491 634 raft_consensus.cc:2932] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 857b0b25994641599b6e146cec879443 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } } }
I20240502 07:59:29.344632 501 raft_consensus.cc:2932] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 857b0b25994641599b6e146cec879443 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } } }
I20240502 07:59:29.346588 32696 catalog_manager.cc:5484] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd reported cstate change: config changed from index -1 to 4, NON_VOTER 857b0b25994641599b6e146cec879443 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "2ca73afa63a04c378a0948e9691601fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:29.355909 634 consensus_queue.cc:237] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } }
I20240502 07:59:29.357388 501 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:29.357455 371 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:59:29.357578 568 consensus_peers.cc:485] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec -> Peer 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132:33885): Couldn't send request to peer 468fd6f1f0fb46ce87ab94f922e89661. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42d23c677d594a6693a052ca2adc2de8. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:29.357789 1848 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:29.357956 1847 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:29.359692 1102 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } }
I20240502 07:59:29.360311 371 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } }
I20240502 07:59:29.360474 501 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } }
I20240502 07:59:29.362771 32695 catalog_manager.cc:5484] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index -1 to 4, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:29.474857 1564 heartbeater.cc:503] Master 127.18.13.190:43401 was elected leader, sending a full tablet report...
I20240502 07:59:29.659191 1695 heartbeater.cc:503] Master 127.18.13.190:43401 was elected leader, sending a full tablet report...
I20240502 07:59:29.673339 1977 ts_tablet_manager.cc:902] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Initiating tablet copy from peer 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:29.674110 1977 tablet_copy_client.cc:308] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:37189
I20240502 07:59:29.677982 655 tablet_copy_service.cc:140] P 733b523e2f3e43539b480b034726d7ec: Received BeginTabletCopySession request for tablet e199f4ff646b4992ab7c6d5527ac32db from peer 783ed3162fe14b76b9cb7a7a6b8bdaa9 ({username='slave'} at 127.18.13.134:41773)
I20240502 07:59:29.678136 655 tablet_copy_service.cc:161] P 733b523e2f3e43539b480b034726d7ec: Beginning new tablet copy session on tablet e199f4ff646b4992ab7c6d5527ac32db from peer 783ed3162fe14b76b9cb7a7a6b8bdaa9 at {username='slave'} at 127.18.13.134:41773: session id = 783ed3162fe14b76b9cb7a7a6b8bdaa9-e199f4ff646b4992ab7c6d5527ac32db
I20240502 07:59:29.679388 655 tablet_copy_source_session.cc:205] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:29.682729 1977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e199f4ff646b4992ab7c6d5527ac32db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:29.688917 1977 tablet_copy_client.cc:783] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:29.689141 1977 tablet_copy_client.cc:647] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:29.690822 1977 tablet_copy_client.cc:523] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:29.692363 1977 tablet_bootstrap.cc:492] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Bootstrap starting.
I20240502 07:59:29.697577 1977 log.cc:824] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:29.717569 1977 tablet_bootstrap.cc:492] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:29.717912 1977 tablet_bootstrap.cc:492] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Bootstrap complete.
I20240502 07:59:29.718195 1977 ts_tablet_manager.cc:1367] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Time spent bootstrapping tablet: real 0.026s user 0.020s sys 0.004s
I20240502 07:59:29.720507 1977 raft_consensus.cc:348] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } }
I20240502 07:59:29.720762 1977 raft_consensus.cc:729] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 783ed3162fe14b76b9cb7a7a6b8bdaa9, State: Initialized, Role: LEARNER
I20240502 07:59:29.721066 1977 consensus_queue.cc:260] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } }
I20240502 07:59:29.721927 1977 ts_tablet_manager.cc:1398] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 07:59:29.722676 655 tablet_copy_service.cc:342] P 733b523e2f3e43539b480b034726d7ec: Request end of tablet copy session 783ed3162fe14b76b9cb7a7a6b8bdaa9-e199f4ff646b4992ab7c6d5527ac32db received from {username='slave'} at 127.18.13.134:41773
I20240502 07:59:29.722760 655 tablet_copy_service.cc:433] P 733b523e2f3e43539b480b034726d7ec: ending tablet copy session 783ed3162fe14b76b9cb7a7a6b8bdaa9-e199f4ff646b4992ab7c6d5527ac32db on tablet e199f4ff646b4992ab7c6d5527ac32db with peer 783ed3162fe14b76b9cb7a7a6b8bdaa9
I20240502 07:59:29.758013 1985 ts_tablet_manager.cc:902] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661: Initiating tablet copy from peer 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:29.758759 1985 tablet_copy_client.cc:308] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:37189
I20240502 07:59:29.762449 655 tablet_copy_service.cc:140] P 733b523e2f3e43539b480b034726d7ec: Received BeginTabletCopySession request for tablet 71841e55040a499bac87b1db55662200 from peer 468fd6f1f0fb46ce87ab94f922e89661 ({username='slave'} at 127.18.13.132:33499)
I20240502 07:59:29.762560 655 tablet_copy_service.cc:161] P 733b523e2f3e43539b480b034726d7ec: Beginning new tablet copy session on tablet 71841e55040a499bac87b1db55662200 from peer 468fd6f1f0fb46ce87ab94f922e89661 at {username='slave'} at 127.18.13.132:33499: session id = 468fd6f1f0fb46ce87ab94f922e89661-71841e55040a499bac87b1db55662200
I20240502 07:59:29.763357 655 tablet_copy_source_session.cc:205] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:29.766378 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71841e55040a499bac87b1db55662200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:29.771993 1985 tablet_copy_client.cc:783] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:29.772207 1985 tablet_copy_client.cc:647] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:29.773850 1985 tablet_copy_client.cc:523] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:29.775872 1985 tablet_bootstrap.cc:492] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661: Bootstrap starting.
I20240502 07:59:29.782951 1985 log.cc:824] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:29.788537 1992 ts_tablet_manager.cc:902] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443: Initiating tablet copy from peer 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429)
I20240502 07:59:29.789307 1992 tablet_copy_client.cc:308] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:46429
I20240502 07:59:29.793630 391 tablet_copy_service.cc:140] P 2ca73afa63a04c378a0948e9691601fd: Received BeginTabletCopySession request for tablet a921e3823fa34be9a26fc4f46e12ef28 from peer 857b0b25994641599b6e146cec879443 ({username='slave'} at 127.18.13.133:42647)
I20240502 07:59:29.793766 391 tablet_copy_service.cc:161] P 2ca73afa63a04c378a0948e9691601fd: Beginning new tablet copy session on tablet a921e3823fa34be9a26fc4f46e12ef28 from peer 857b0b25994641599b6e146cec879443 at {username='slave'} at 127.18.13.133:42647: session id = 857b0b25994641599b6e146cec879443-a921e3823fa34be9a26fc4f46e12ef28
I20240502 07:59:29.794878 391 tablet_copy_source_session.cc:205] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:29.797959 1992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a921e3823fa34be9a26fc4f46e12ef28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:29.803223 1992 tablet_copy_client.cc:783] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:29.803427 1992 tablet_copy_client.cc:647] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:29.805424 1992 tablet_copy_client.cc:523] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:29.805682 1985 tablet_bootstrap.cc:492] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:29.806011 1985 tablet_bootstrap.cc:492] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661: Bootstrap complete.
I20240502 07:59:29.806288 1985 ts_tablet_manager.cc:1367] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661: Time spent bootstrapping tablet: real 0.031s user 0.030s sys 0.000s
I20240502 07:59:29.807044 1992 tablet_bootstrap.cc:492] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443: Bootstrap starting.
I20240502 07:59:29.808334 1985 raft_consensus.cc:348] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } }
I20240502 07:59:29.808548 1985 raft_consensus.cc:729] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 468fd6f1f0fb46ce87ab94f922e89661, State: Initialized, Role: LEARNER
I20240502 07:59:29.808857 1985 consensus_queue.cc:260] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } }
I20240502 07:59:29.809641 1985 ts_tablet_manager.cc:1398] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:29.810299 655 tablet_copy_service.cc:342] P 733b523e2f3e43539b480b034726d7ec: Request end of tablet copy session 468fd6f1f0fb46ce87ab94f922e89661-71841e55040a499bac87b1db55662200 received from {username='slave'} at 127.18.13.132:33499
I20240502 07:59:29.810395 655 tablet_copy_service.cc:433] P 733b523e2f3e43539b480b034726d7ec: ending tablet copy session 468fd6f1f0fb46ce87ab94f922e89661-71841e55040a499bac87b1db55662200 on tablet 71841e55040a499bac87b1db55662200 with peer 468fd6f1f0fb46ce87ab94f922e89661
I20240502 07:59:29.812987 1992 log.cc:824] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:29.825748 1985 ts_tablet_manager.cc:902] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661: Initiating tablet copy from peer 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:29.826253 1985 tablet_copy_client.cc:308] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:37189
I20240502 07:59:29.826771 655 tablet_copy_service.cc:140] P 733b523e2f3e43539b480b034726d7ec: Received BeginTabletCopySession request for tablet cc751cb753a840d28fb8787ca1ecafb0 from peer 468fd6f1f0fb46ce87ab94f922e89661 ({username='slave'} at 127.18.13.132:33499)
I20240502 07:59:29.826875 655 tablet_copy_service.cc:161] P 733b523e2f3e43539b480b034726d7ec: Beginning new tablet copy session on tablet cc751cb753a840d28fb8787ca1ecafb0 from peer 468fd6f1f0fb46ce87ab94f922e89661 at {username='slave'} at 127.18.13.132:33499: session id = 468fd6f1f0fb46ce87ab94f922e89661-cc751cb753a840d28fb8787ca1ecafb0
I20240502 07:59:29.827616 655 tablet_copy_source_session.cc:205] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:29.828521 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc751cb753a840d28fb8787ca1ecafb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:29.830348 1985 tablet_copy_client.cc:783] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:29.830502 1985 tablet_copy_client.cc:647] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:29.831705 1985 tablet_copy_client.cc:523] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:29.832929 1985 tablet_bootstrap.cc:492] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661: Bootstrap starting.
I20240502 07:59:29.833930 1992 tablet_bootstrap.cc:492] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:29.834297 1992 tablet_bootstrap.cc:492] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443: Bootstrap complete.
I20240502 07:59:29.834641 1992 ts_tablet_manager.cc:1367] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443: Time spent bootstrapping tablet: real 0.028s user 0.026s sys 0.000s
I20240502 07:59:29.837226 1992 raft_consensus.cc:348] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } }
I20240502 07:59:29.837538 1992 raft_consensus.cc:729] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 857b0b25994641599b6e146cec879443, State: Initialized, Role: LEARNER
I20240502 07:59:29.837980 1992 consensus_queue.cc:260] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } }
I20240502 07:59:29.838959 1992 ts_tablet_manager.cc:1398] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20240502 07:59:29.839830 391 tablet_copy_service.cc:342] P 2ca73afa63a04c378a0948e9691601fd: Request end of tablet copy session 857b0b25994641599b6e146cec879443-a921e3823fa34be9a26fc4f46e12ef28 received from {username='slave'} at 127.18.13.133:42647
I20240502 07:59:29.839926 391 tablet_copy_service.cc:433] P 2ca73afa63a04c378a0948e9691601fd: ending tablet copy session 857b0b25994641599b6e146cec879443-a921e3823fa34be9a26fc4f46e12ef28 on tablet a921e3823fa34be9a26fc4f46e12ef28 with peer 857b0b25994641599b6e146cec879443
I20240502 07:59:29.842411 1992 ts_tablet_manager.cc:902] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443: Initiating tablet copy from peer 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429)
I20240502 07:59:29.842804 1992 tablet_copy_client.cc:308] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:46429
I20240502 07:59:29.843195 391 tablet_copy_service.cc:140] P 2ca73afa63a04c378a0948e9691601fd: Received BeginTabletCopySession request for tablet b51370efd02a42ec920c531abf4834b9 from peer 857b0b25994641599b6e146cec879443 ({username='slave'} at 127.18.13.133:42647)
I20240502 07:59:29.843312 391 tablet_copy_service.cc:161] P 2ca73afa63a04c378a0948e9691601fd: Beginning new tablet copy session on tablet b51370efd02a42ec920c531abf4834b9 from peer 857b0b25994641599b6e146cec879443 at {username='slave'} at 127.18.13.133:42647: session id = 857b0b25994641599b6e146cec879443-b51370efd02a42ec920c531abf4834b9
I20240502 07:59:29.844236 391 tablet_copy_source_session.cc:205] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:29.845134 1992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b51370efd02a42ec920c531abf4834b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:29.847661 1992 tablet_copy_client.cc:783] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:29.847877 1992 tablet_copy_client.cc:647] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:29.849050 1992 tablet_copy_client.cc:523] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:29.850358 1992 tablet_bootstrap.cc:492] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443: Bootstrap starting.
I20240502 07:59:29.859201 1985 tablet_bootstrap.cc:492] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:29.859562 1985 tablet_bootstrap.cc:492] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661: Bootstrap complete.
I20240502 07:59:29.859778 1985 ts_tablet_manager.cc:1367] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661: Time spent bootstrapping tablet: real 0.027s user 0.027s sys 0.000s
I20240502 07:59:29.860411 1985 raft_consensus.cc:348] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } }
I20240502 07:59:29.860548 1985 raft_consensus.cc:729] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 468fd6f1f0fb46ce87ab94f922e89661, State: Initialized, Role: LEARNER
I20240502 07:59:29.860682 1985 consensus_queue.cc:260] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } }
I20240502 07:59:29.860980 1985 ts_tablet_manager.cc:1398] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:29.861923 655 tablet_copy_service.cc:342] P 733b523e2f3e43539b480b034726d7ec: Request end of tablet copy session 468fd6f1f0fb46ce87ab94f922e89661-cc751cb753a840d28fb8787ca1ecafb0 received from {username='slave'} at 127.18.13.132:33499
I20240502 07:59:29.862002 655 tablet_copy_service.cc:433] P 733b523e2f3e43539b480b034726d7ec: ending tablet copy session 468fd6f1f0fb46ce87ab94f922e89661-cc751cb753a840d28fb8787ca1ecafb0 on tablet cc751cb753a840d28fb8787ca1ecafb0 with peer 468fd6f1f0fb46ce87ab94f922e89661
I20240502 07:59:29.876444 1992 tablet_bootstrap.cc:492] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:29.876752 1992 tablet_bootstrap.cc:492] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443: Bootstrap complete.
I20240502 07:59:29.876976 1992 ts_tablet_manager.cc:1367] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443: Time spent bootstrapping tablet: real 0.027s user 0.025s sys 0.000s
I20240502 07:59:29.877599 1992 raft_consensus.cc:348] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } }
I20240502 07:59:29.877744 1992 raft_consensus.cc:729] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 857b0b25994641599b6e146cec879443, State: Initialized, Role: LEARNER
I20240502 07:59:29.877887 1992 consensus_queue.cc:260] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } }
I20240502 07:59:29.878253 1992 ts_tablet_manager.cc:1398] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:29.878914 391 tablet_copy_service.cc:342] P 2ca73afa63a04c378a0948e9691601fd: Request end of tablet copy session 857b0b25994641599b6e146cec879443-b51370efd02a42ec920c531abf4834b9 received from {username='slave'} at 127.18.13.133:42647
I20240502 07:59:29.878994 391 tablet_copy_service.cc:433] P 2ca73afa63a04c378a0948e9691601fd: ending tablet copy session 857b0b25994641599b6e146cec879443-b51370efd02a42ec920c531abf4834b9 on tablet b51370efd02a42ec920c531abf4834b9 with peer 857b0b25994641599b6e146cec879443
I20240502 07:59:29.887126 1977 ts_tablet_manager.cc:902] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Initiating tablet copy from peer 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429)
I20240502 07:59:29.887634 1977 tablet_copy_client.cc:308] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:46429
I20240502 07:59:29.890481 391 tablet_copy_service.cc:140] P 2ca73afa63a04c378a0948e9691601fd: Received BeginTabletCopySession request for tablet 590180286acf49189ba03c8725946565 from peer 783ed3162fe14b76b9cb7a7a6b8bdaa9 ({username='slave'} at 127.18.13.134:46323)
I20240502 07:59:29.890574 391 tablet_copy_service.cc:161] P 2ca73afa63a04c378a0948e9691601fd: Beginning new tablet copy session on tablet 590180286acf49189ba03c8725946565 from peer 783ed3162fe14b76b9cb7a7a6b8bdaa9 at {username='slave'} at 127.18.13.134:46323: session id = 783ed3162fe14b76b9cb7a7a6b8bdaa9-590180286acf49189ba03c8725946565
I20240502 07:59:29.891254 391 tablet_copy_source_session.cc:205] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:29.892047 1977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590180286acf49189ba03c8725946565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:29.893844 1977 tablet_copy_client.cc:783] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:29.894011 1977 tablet_copy_client.cc:647] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:29.895057 1977 tablet_copy_client.cc:523] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:29.896092 1977 tablet_bootstrap.cc:492] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Bootstrap starting.
I20240502 07:59:29.904065 1985 ts_tablet_manager.cc:902] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661: Initiating tablet copy from peer 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:29.904590 1985 tablet_copy_client.cc:308] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:37189
I20240502 07:59:29.905138 655 tablet_copy_service.cc:140] P 733b523e2f3e43539b480b034726d7ec: Received BeginTabletCopySession request for tablet 42d23c677d594a6693a052ca2adc2de8 from peer 468fd6f1f0fb46ce87ab94f922e89661 ({username='slave'} at 127.18.13.132:33499)
I20240502 07:59:29.905257 655 tablet_copy_service.cc:161] P 733b523e2f3e43539b480b034726d7ec: Beginning new tablet copy session on tablet 42d23c677d594a6693a052ca2adc2de8 from peer 468fd6f1f0fb46ce87ab94f922e89661 at {username='slave'} at 127.18.13.132:33499: session id = 468fd6f1f0fb46ce87ab94f922e89661-42d23c677d594a6693a052ca2adc2de8
I20240502 07:59:29.905993 655 tablet_copy_source_session.cc:205] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:29.906878 1985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d23c677d594a6693a052ca2adc2de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:29.909147 1985 tablet_copy_client.cc:783] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:29.909307 1985 tablet_copy_client.cc:647] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:29.910758 1985 tablet_copy_client.cc:523] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:29.911947 1985 tablet_bootstrap.cc:492] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661: Bootstrap starting.
I20240502 07:59:29.916389 1977 tablet_bootstrap.cc:492] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:29.916661 1977 tablet_bootstrap.cc:492] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Bootstrap complete.
I20240502 07:59:29.916839 1977 ts_tablet_manager.cc:1367] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.000s
I20240502 07:59:29.917351 1977 raft_consensus.cc:348] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } }
I20240502 07:59:29.917460 1977 raft_consensus.cc:729] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 783ed3162fe14b76b9cb7a7a6b8bdaa9, State: Initialized, Role: LEARNER
I20240502 07:59:29.917549 1977 consensus_queue.cc:260] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } }
I20240502 07:59:29.917829 1977 ts_tablet_manager.cc:1398] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:29.918394 391 tablet_copy_service.cc:342] P 2ca73afa63a04c378a0948e9691601fd: Request end of tablet copy session 783ed3162fe14b76b9cb7a7a6b8bdaa9-590180286acf49189ba03c8725946565 received from {username='slave'} at 127.18.13.134:46323
I20240502 07:59:29.918494 391 tablet_copy_service.cc:433] P 2ca73afa63a04c378a0948e9691601fd: ending tablet copy session 783ed3162fe14b76b9cb7a7a6b8bdaa9-590180286acf49189ba03c8725946565 on tablet 590180286acf49189ba03c8725946565 with peer 783ed3162fe14b76b9cb7a7a6b8bdaa9
I20240502 07:59:29.933176 1985 tablet_bootstrap.cc:492] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:29.933516 1985 tablet_bootstrap.cc:492] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661: Bootstrap complete.
I20240502 07:59:29.933703 1985 ts_tablet_manager.cc:1367] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.000s
I20240502 07:59:29.934113 1985 raft_consensus.cc:348] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } }
I20240502 07:59:29.934222 1985 raft_consensus.cc:729] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 468fd6f1f0fb46ce87ab94f922e89661, State: Initialized, Role: LEARNER
I20240502 07:59:29.934311 1985 consensus_queue.cc:260] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } }
I20240502 07:59:29.934602 1985 ts_tablet_manager.cc:1398] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:29.935272 655 tablet_copy_service.cc:342] P 733b523e2f3e43539b480b034726d7ec: Request end of tablet copy session 468fd6f1f0fb46ce87ab94f922e89661-42d23c677d594a6693a052ca2adc2de8 received from {username='slave'} at 127.18.13.132:33499
I20240502 07:59:29.935375 655 tablet_copy_service.cc:433] P 733b523e2f3e43539b480b034726d7ec: ending tablet copy session 468fd6f1f0fb46ce87ab94f922e89661-42d23c677d594a6693a052ca2adc2de8 on tablet 42d23c677d594a6693a052ca2adc2de8 with peer 468fd6f1f0fb46ce87ab94f922e89661
I20240502 07:59:30.071283 1516 raft_consensus.cc:1203] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:30.176548 1649 raft_consensus.cc:1203] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:30.204815 1387 raft_consensus.cc:1203] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:30.309085 1387 raft_consensus.cc:1203] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:30.311237 1516 raft_consensus.cc:1203] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:30.326709 1387 raft_consensus.cc:1203] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:30.364173 1649 raft_consensus.cc:1203] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:30.595170 2056 raft_consensus.cc:1050] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec: attempting to promote NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 to VOTER
I20240502 07:59:30.595746 2056 consensus_queue.cc:237] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } }
I20240502 07:59:30.597280 1516 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.597296 371 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.597694 501 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.597700 2151 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.598232 2057 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.598456 1848 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.600718 2057 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.601212 501 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.602191 1516 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.603500 370 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.604238 32692 catalog_manager.cc:5484] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index 4 to 5, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:30.608386 634 consensus_queue.cc:237] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } }
I20240502 07:59:30.609712 1516 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:30.610167 370 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:30.610244 2151 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:30.610577 1848 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:30.612211 2151 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.612645 371 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.613869 32683 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e199f4ff646b4992ab7c6d5527ac32db with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:30.614656 32688 catalog_manager.cc:5484] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index 5 to 6, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:30.614789 1516 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.617679 483 tablet_service.cc:1514] Processing DeleteTablet for tablet e199f4ff646b4992ab7c6d5527ac32db with delete_type TABLET_DATA_TOMBSTONED (TS 5a60a9fab026458293b88d08fb10e88a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36396
I20240502 07:59:30.618242 2169 tablet_replica.cc:330] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: stopping tablet replica
I20240502 07:59:30.618947 2169 raft_consensus.cc:2229] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:30.619216 2169 raft_consensus.cc:2258] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:30.620456 2169 ts_tablet_manager.cc:1875] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:30.623137 2169 ts_tablet_manager.cc:1888] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:30.623263 2169 log.cc:1196] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/e199f4ff646b4992ab7c6d5527ac32db
I20240502 07:59:30.623955 32681 catalog_manager.cc:4830] TS 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457): tablet e199f4ff646b4992ab7c6d5527ac32db (table rebalance_test_table_2 [id=63eb7086af244504b9daf9beaeb2a9bd]) successfully deleted
I20240502 07:59:30.647336 1848 raft_consensus.cc:1050] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec: attempting to promote NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 to VOTER
I20240502 07:59:30.647735 1848 consensus_queue.cc:237] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } }
I20240502 07:59:30.649410 501 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.649492 1387 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.649502 371 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.649996 1848 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.650183 2057 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.650329 2151 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.653928 2057 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.655582 371 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.656036 501 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.656270 32696 catalog_manager.cc:5484] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index 4 to 5, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:30.657114 1387 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.661247 634 consensus_queue.cc:237] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } }
I20240502 07:59:30.662420 1387 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:30.662487 501 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:30.662791 2057 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:30.662956 1848 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:30.664572 2151 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.664997 1387 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.664983 501 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.666121 32683 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cc751cb753a840d28fb8787ca1ecafb0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:30.666822 32695 catalog_manager.cc:5484] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index 5 to 6, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:30.669553 349 tablet_service.cc:1514] Processing DeleteTablet for tablet cc751cb753a840d28fb8787ca1ecafb0 with delete_type TABLET_DATA_TOMBSTONED (TS 2ca73afa63a04c378a0948e9691601fd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54562
I20240502 07:59:30.670024 2179 tablet_replica.cc:330] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: stopping tablet replica
I20240502 07:59:30.670317 2179 raft_consensus.cc:2229] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:30.670526 2179 raft_consensus.cc:2258] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:30.671530 2179 ts_tablet_manager.cc:1875] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:30.673488 2179 ts_tablet_manager.cc:1888] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:30.673601 2179 log.cc:1196] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/cc751cb753a840d28fb8787ca1ecafb0
I20240502 07:59:30.674122 32681 catalog_manager.cc:4830] TS 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429): tablet cc751cb753a840d28fb8787ca1ecafb0 (table rebalance_test_table_0 [id=442b95cfe4e84e99965954832dd504ff]) successfully deleted
I20240502 07:59:30.715581 1848 raft_consensus.cc:1050] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec: attempting to promote NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 to VOTER
I20240502 07:59:30.716006 1848 consensus_queue.cc:237] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } }
I20240502 07:59:30.717476 1387 raft_consensus.cc:1261] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.717476 501 raft_consensus.cc:1261] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.717487 371 raft_consensus.cc:1261] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.717901 2056 consensus_queue.cc:1035] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.718066 1848 consensus_queue.cc:1035] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.718196 2057 consensus_queue.cc:1035] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.720258 2057 raft_consensus.cc:2932] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.720752 1387 raft_consensus.cc:2932] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.720784 501 raft_consensus.cc:2932] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.722515 371 raft_consensus.cc:2932] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.723235 32696 catalog_manager.cc:5484] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index 4 to 5, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:30.726243 634 consensus_queue.cc:237] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } }
I20240502 07:59:30.727515 1387 raft_consensus.cc:1261] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:30.727653 501 raft_consensus.cc:1261] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:30.728140 1848 consensus_queue.cc:1035] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:30.728302 2056 consensus_queue.cc:1035] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:30.730011 2151 raft_consensus.cc:2932] T 71841e55040a499bac87b1db55662200 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.730392 501 raft_consensus.cc:2932] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.730449 1387 raft_consensus.cc:2932] T 71841e55040a499bac87b1db55662200 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.731603 32683 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71841e55040a499bac87b1db55662200 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:30.732612 32688 catalog_manager.cc:5484] T 71841e55040a499bac87b1db55662200 P 5a60a9fab026458293b88d08fb10e88a reported cstate change: config changed from index 5 to 6, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.734900 349 tablet_service.cc:1514] Processing DeleteTablet for tablet 71841e55040a499bac87b1db55662200 with delete_type TABLET_DATA_TOMBSTONED (TS 2ca73afa63a04c378a0948e9691601fd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54562
I20240502 07:59:30.735126 2179 tablet_replica.cc:330] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd: stopping tablet replica
I20240502 07:59:30.735245 2179 raft_consensus.cc:2229] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:30.735363 2179 raft_consensus.cc:2258] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:30.736066 2179 ts_tablet_manager.cc:1875] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:30.737795 2179 ts_tablet_manager.cc:1888] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:30.737864 2179 log.cc:1196] T 71841e55040a499bac87b1db55662200 P 2ca73afa63a04c378a0948e9691601fd: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/71841e55040a499bac87b1db55662200
I20240502 07:59:30.738392 32681 catalog_manager.cc:4830] TS 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429): tablet 71841e55040a499bac87b1db55662200 (table rebalance_test_table_1 [id=a353f7ca52eb4a0e86f45854ebf5dbe6]) successfully deleted
I20240502 07:59:30.746011 1848 raft_consensus.cc:1050] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec: attempting to promote NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 to VOTER
I20240502 07:59:30.746433 1848 consensus_queue.cc:237] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } }
I20240502 07:59:30.747721 371 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.747812 1387 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.747923 501 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.748119 1848 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.748368 2151 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.748529 2057 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.750188 2057 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.750669 371 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.750985 1387 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.752393 501 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.753180 32692 catalog_manager.cc:5484] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd reported cstate change: config changed from index 4 to 5, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.756804 634 consensus_queue.cc:237] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } }
I20240502 07:59:30.757863 501 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:30.757889 1387 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:30.758312 1848 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:30.758497 2057 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:30.759934 1848 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.760352 501 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.760406 1387 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.761574 32683 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 42d23c677d594a6693a052ca2adc2de8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:30.762707 32696 catalog_manager.cc:5484] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a reported cstate change: config changed from index 5 to 6, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } }
I20240502 07:59:30.764847 349 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d23c677d594a6693a052ca2adc2de8 with delete_type TABLET_DATA_TOMBSTONED (TS 2ca73afa63a04c378a0948e9691601fd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54562
I20240502 07:59:30.765120 2179 tablet_replica.cc:330] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd: stopping tablet replica
I20240502 07:59:30.765269 2179 raft_consensus.cc:2229] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:30.765447 2179 raft_consensus.cc:2258] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:30.766121 2179 ts_tablet_manager.cc:1875] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:30.767760 2179 ts_tablet_manager.cc:1888] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:30.767864 2179 log.cc:1196] T 42d23c677d594a6693a052ca2adc2de8 P 2ca73afa63a04c378a0948e9691601fd: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/42d23c677d594a6693a052ca2adc2de8
I20240502 07:59:30.768406 32681 catalog_manager.cc:4830] TS 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429): tablet 42d23c677d594a6693a052ca2adc2de8 (table rebalance_test_table_0 [id=442b95cfe4e84e99965954832dd504ff]) successfully deleted
I20240502 07:59:30.768512 1870 raft_consensus.cc:1050] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd: attempting to promote NON_VOTER 857b0b25994641599b6e146cec879443 to VOTER
I20240502 07:59:30.768916 1870 consensus_queue.cc:237] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } }
I20240502 07:59:30.770198 634 raft_consensus.cc:1261] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.770515 1649 raft_consensus.cc:1261] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443 [term 1 LEARNER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.770617 2189 consensus_queue.cc:1035] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.770942 501 raft_consensus.cc:1261] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.770960 2094 consensus_queue.cc:1035] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.771337 2094 consensus_queue.cc:1035] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.773366 2189 raft_consensus.cc:2932] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 857b0b25994641599b6e146cec879443 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.773805 501 raft_consensus.cc:2932] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 857b0b25994641599b6e146cec879443 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.775385 634 raft_consensus.cc:2932] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 857b0b25994641599b6e146cec879443 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.776037 32688 catalog_manager.cc:5484] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a reported cstate change: config changed from index 4 to 5, 857b0b25994641599b6e146cec879443 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2ca73afa63a04c378a0948e9691601fd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.782953 1649 raft_consensus.cc:2932] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 857b0b25994641599b6e146cec879443 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.784392 371 consensus_queue.cc:237] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } }
I20240502 07:59:30.785857 501 raft_consensus.cc:1261] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:30.786461 1648 raft_consensus.cc:1261] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443 [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:30.786960 2094 consensus_queue.cc:1035] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:30.787256 2094 consensus_queue.cc:1035] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:30.788873 2094 raft_consensus.cc:2932] T a921e3823fa34be9a26fc4f46e12ef28 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 733b523e2f3e43539b480b034726d7ec (127.18.13.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.789317 501 raft_consensus.cc:2932] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 733b523e2f3e43539b480b034726d7ec (127.18.13.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.790234 1648 raft_consensus.cc:2932] T a921e3823fa34be9a26fc4f46e12ef28 P 857b0b25994641599b6e146cec879443 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 733b523e2f3e43539b480b034726d7ec (127.18.13.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.790390 32681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a921e3823fa34be9a26fc4f46e12ef28 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:30.792431 32695 catalog_manager.cc:5484] T a921e3823fa34be9a26fc4f46e12ef28 P 5a60a9fab026458293b88d08fb10e88a reported cstate change: config changed from index 5 to 6, VOTER 733b523e2f3e43539b480b034726d7ec (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "2ca73afa63a04c378a0948e9691601fd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.794864 614 tablet_service.cc:1514] Processing DeleteTablet for tablet a921e3823fa34be9a26fc4f46e12ef28 with delete_type TABLET_DATA_TOMBSTONED (TS 733b523e2f3e43539b480b034726d7ec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56412
I20240502 07:59:30.795410 2193 tablet_replica.cc:330] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec: stopping tablet replica
I20240502 07:59:30.795682 2193 raft_consensus.cc:2229] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:30.795939 2193 raft_consensus.cc:2258] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:30.796995 2193 ts_tablet_manager.cc:1875] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:30.798890 2193 ts_tablet_manager.cc:1888] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:30.798982 2193 log.cc:1196] T a921e3823fa34be9a26fc4f46e12ef28 P 733b523e2f3e43539b480b034726d7ec: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/a921e3823fa34be9a26fc4f46e12ef28
I20240502 07:59:30.799517 32683 catalog_manager.cc:4830] TS 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189): tablet a921e3823fa34be9a26fc4f46e12ef28 (table rebalance_test_table_1 [id=a353f7ca52eb4a0e86f45854ebf5dbe6]) successfully deleted
I20240502 07:59:30.818780 2093 raft_consensus.cc:1050] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd: attempting to promote NON_VOTER 857b0b25994641599b6e146cec879443 to VOTER
I20240502 07:59:30.819198 2093 consensus_queue.cc:237] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } }
I20240502 07:59:30.820698 1648 raft_consensus.cc:1261] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443 [term 1 LEARNER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.820698 634 raft_consensus.cc:1261] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.820712 501 raft_consensus.cc:1261] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.821090 2189 consensus_queue.cc:1035] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.821327 1870 consensus_queue.cc:1035] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.821506 2093 consensus_queue.cc:1035] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.823345 1870 raft_consensus.cc:2932] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 857b0b25994641599b6e146cec879443 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.823801 634 raft_consensus.cc:2932] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 857b0b25994641599b6e146cec879443 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.823968 501 raft_consensus.cc:2932] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 857b0b25994641599b6e146cec879443 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.826151 1648 raft_consensus.cc:2932] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 857b0b25994641599b6e146cec879443 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.826422 32696 catalog_manager.cc:5484] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a reported cstate change: config changed from index 4 to 5, 857b0b25994641599b6e146cec879443 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2ca73afa63a04c378a0948e9691601fd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.830941 371 consensus_queue.cc:237] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } }
I20240502 07:59:30.832366 1648 raft_consensus.cc:1261] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443 [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:30.832749 501 raft_consensus.cc:1261] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:30.833086 2189 consensus_queue.cc:1035] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:30.833313 1870 consensus_queue.cc:1035] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:30.835745 2189 raft_consensus.cc:2932] T b51370efd02a42ec920c531abf4834b9 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 733b523e2f3e43539b480b034726d7ec (127.18.13.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.836257 501 raft_consensus.cc:2932] T b51370efd02a42ec920c531abf4834b9 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 733b523e2f3e43539b480b034726d7ec (127.18.13.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.836637 1648 raft_consensus.cc:2932] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 733b523e2f3e43539b480b034726d7ec (127.18.13.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.838640 32681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b51370efd02a42ec920c531abf4834b9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:30.839001 32688 catalog_manager.cc:5484] T b51370efd02a42ec920c531abf4834b9 P 857b0b25994641599b6e146cec879443 reported cstate change: config changed from index 5 to 6, VOTER 733b523e2f3e43539b480b034726d7ec (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "2ca73afa63a04c378a0948e9691601fd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:30.841720 614 tablet_service.cc:1514] Processing DeleteTablet for tablet b51370efd02a42ec920c531abf4834b9 with delete_type TABLET_DATA_TOMBSTONED (TS 733b523e2f3e43539b480b034726d7ec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56412
I20240502 07:59:30.841969 2193 tablet_replica.cc:330] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec: stopping tablet replica
I20240502 07:59:30.842113 2193 raft_consensus.cc:2229] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:30.842293 2193 raft_consensus.cc:2258] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:30.842921 2193 ts_tablet_manager.cc:1875] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:30.844942 2193 ts_tablet_manager.cc:1888] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:30.845021 2193 log.cc:1196] T b51370efd02a42ec920c531abf4834b9 P 733b523e2f3e43539b480b034726d7ec: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/b51370efd02a42ec920c531abf4834b9
I20240502 07:59:30.845589 32683 catalog_manager.cc:4830] TS 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189): tablet b51370efd02a42ec920c531abf4834b9 (table rebalance_test_table_0 [id=442b95cfe4e84e99965954832dd504ff]) successfully deleted
I20240502 07:59:30.887713 2094 raft_consensus.cc:1050] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd: attempting to promote NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 to VOTER
I20240502 07:59:30.888242 2094 consensus_queue.cc:237] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } }
I20240502 07:59:30.889664 1516 raft_consensus.cc:1261] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.889691 634 raft_consensus.cc:1261] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.889853 501 raft_consensus.cc:1261] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:30.890054 2189 consensus_queue.cc:1035] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.890244 2093 consensus_queue.cc:1035] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.890393 2094 consensus_queue.cc:1035] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:30.892266 2093 raft_consensus.cc:2932] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.892706 1516 raft_consensus.cc:2932] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.894042 634 raft_consensus.cc:2932] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.894771 32692 catalog_manager.cc:5484] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 reported cstate change: config changed from index 4 to 5, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2ca73afa63a04c378a0948e9691601fd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.896255 501 raft_consensus.cc:2932] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.898315 371 consensus_queue.cc:237] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } }
I20240502 07:59:30.899732 634 raft_consensus.cc:1261] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:30.900133 2094 consensus_queue.cc:1035] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:30.900332 1516 raft_consensus.cc:1261] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Refusing update from remote peer 2ca73afa63a04c378a0948e9691601fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:30.900671 2094 consensus_queue.cc:1035] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:30.901885 2093 raft_consensus.cc:2932] T 590180286acf49189ba03c8725946565 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.902563 634 raft_consensus.cc:2932] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.902662 1516 raft_consensus.cc:2932] T 590180286acf49189ba03c8725946565 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.903288 32681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 590180286acf49189ba03c8725946565 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:30.904803 32692 catalog_manager.cc:5484] T 590180286acf49189ba03c8725946565 P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index 5 to 6, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "2ca73afa63a04c378a0948e9691601fd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:30.907173 483 tablet_service.cc:1514] Processing DeleteTablet for tablet 590180286acf49189ba03c8725946565 with delete_type TABLET_DATA_TOMBSTONED (TS 5a60a9fab026458293b88d08fb10e88a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36396
I20240502 07:59:30.907405 2169 tablet_replica.cc:330] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a: stopping tablet replica
I20240502 07:59:30.907532 2169 raft_consensus.cc:2229] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:30.907656 2169 raft_consensus.cc:2258] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:30.908275 2169 ts_tablet_manager.cc:1875] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:30.909945 2169 ts_tablet_manager.cc:1888] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:30.910018 2169 log.cc:1196] T 590180286acf49189ba03c8725946565 P 5a60a9fab026458293b88d08fb10e88a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/590180286acf49189ba03c8725946565
I20240502 07:59:30.910626 32681 catalog_manager.cc:4830] TS 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457): tablet 590180286acf49189ba03c8725946565 (table rebalance_test_table_2 [id=63eb7086af244504b9daf9beaeb2a9bd]) successfully deleted
I20240502 07:59:31.142144 349 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:31.142812 614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:31.143949 1629 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:31.148962 483 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:31.154558 1498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:31.155714 1367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:31.192648 634 consensus_queue.cc:237] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } }
I20240502 07:59:31.194108 371 raft_consensus.cc:1261] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:31.194190 501 raft_consensus.cc:1261] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:59:31.194231 569 consensus_peers.cc:485] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec -> Peer 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134:39939): Couldn't send request to peer 783ed3162fe14b76b9cb7a7a6b8bdaa9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bf16ec8c5854b2daaccabe49d10ba26. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:31.194577 2057 consensus_queue.cc:1035] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:31.194766 2151 consensus_queue.cc:1035] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:31.196483 2056 raft_consensus.cc:2932] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } } }
I20240502 07:59:31.196949 371 raft_consensus.cc:2932] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } } }
I20240502 07:59:31.197007 501 raft_consensus.cc:2932] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } } }
I20240502 07:59:31.199299 32695 catalog_manager.cc:5484] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index -1 to 4, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:31.208263 501 consensus_queue.cc:237] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } }
I20240502 07:59:31.209929 634 raft_consensus.cc:1261] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Refusing update from remote peer 5a60a9fab026458293b88d08fb10e88a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:31.210202 371 raft_consensus.cc:1261] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 5a60a9fab026458293b88d08fb10e88a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:31.210322 2295 consensus_queue.cc:1035] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:31.210704 1229 consensus_queue.cc:1035] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:31.212491 1229 raft_consensus.cc:2932] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 857b0b25994641599b6e146cec879443 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } } }
I20240502 07:59:31.212781 634 raft_consensus.cc:2932] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 857b0b25994641599b6e146cec879443 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } } }
I20240502 07:59:31.215297 32692 catalog_manager.cc:5484] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index -1 to 4, NON_VOTER 857b0b25994641599b6e146cec879443 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "5a60a9fab026458293b88d08fb10e88a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } } }
I20240502 07:59:31.215595 371 raft_consensus.cc:2932] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 857b0b25994641599b6e146cec879443 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } } }
W20240502 07:59:31.216122 436 consensus_peers.cc:485] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a -> Peer 857b0b25994641599b6e146cec879443 (127.18.13.133:44143): Couldn't send request to peer 857b0b25994641599b6e146cec879443. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0d4cda1deb247d1872658e08dbd6d6b. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:31.586179 2319 ts_tablet_manager.cc:902] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Initiating tablet copy from peer 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:31.586709 2319 tablet_copy_client.cc:308] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:37189
I20240502 07:59:31.587265 655 tablet_copy_service.cc:140] P 733b523e2f3e43539b480b034726d7ec: Received BeginTabletCopySession request for tablet 9bf16ec8c5854b2daaccabe49d10ba26 from peer 783ed3162fe14b76b9cb7a7a6b8bdaa9 ({username='slave'} at 127.18.13.134:41773)
I20240502 07:59:31.587369 655 tablet_copy_service.cc:161] P 733b523e2f3e43539b480b034726d7ec: Beginning new tablet copy session on tablet 9bf16ec8c5854b2daaccabe49d10ba26 from peer 783ed3162fe14b76b9cb7a7a6b8bdaa9 at {username='slave'} at 127.18.13.134:41773: session id = 783ed3162fe14b76b9cb7a7a6b8bdaa9-9bf16ec8c5854b2daaccabe49d10ba26
I20240502 07:59:31.588171 655 tablet_copy_source_session.cc:205] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:31.589054 2319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bf16ec8c5854b2daaccabe49d10ba26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:31.591424 2319 tablet_copy_client.cc:783] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:31.591609 2319 tablet_copy_client.cc:647] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:31.592803 2319 tablet_copy_client.cc:523] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:31.593871 2319 tablet_bootstrap.cc:492] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Bootstrap starting.
I20240502 07:59:31.612071 2319 tablet_bootstrap.cc:492] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:31.612349 2319 tablet_bootstrap.cc:492] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Bootstrap complete.
I20240502 07:59:31.612497 2319 ts_tablet_manager.cc:1367] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Time spent bootstrapping tablet: real 0.019s user 0.019s sys 0.000s
I20240502 07:59:31.612895 2319 raft_consensus.cc:348] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } }
I20240502 07:59:31.613001 2319 raft_consensus.cc:729] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 783ed3162fe14b76b9cb7a7a6b8bdaa9, State: Initialized, Role: LEARNER
I20240502 07:59:31.613085 2319 consensus_queue.cc:260] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } }
I20240502 07:59:31.613394 2319 ts_tablet_manager.cc:1398] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:31.614127 655 tablet_copy_service.cc:342] P 733b523e2f3e43539b480b034726d7ec: Request end of tablet copy session 783ed3162fe14b76b9cb7a7a6b8bdaa9-9bf16ec8c5854b2daaccabe49d10ba26 received from {username='slave'} at 127.18.13.134:41773
I20240502 07:59:31.614223 655 tablet_copy_service.cc:433] P 733b523e2f3e43539b480b034726d7ec: ending tablet copy session 783ed3162fe14b76b9cb7a7a6b8bdaa9-9bf16ec8c5854b2daaccabe49d10ba26 on tablet 9bf16ec8c5854b2daaccabe49d10ba26 with peer 783ed3162fe14b76b9cb7a7a6b8bdaa9
I20240502 07:59:31.818954 2336 ts_tablet_manager.cc:902] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443: Initiating tablet copy from peer 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457)
I20240502 07:59:31.819487 2336 tablet_copy_client.cc:308] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:37457
I20240502 07:59:31.823225 523 tablet_copy_service.cc:140] P 5a60a9fab026458293b88d08fb10e88a: Received BeginTabletCopySession request for tablet d0d4cda1deb247d1872658e08dbd6d6b from peer 857b0b25994641599b6e146cec879443 ({username='slave'} at 127.18.13.133:57361)
I20240502 07:59:31.823359 523 tablet_copy_service.cc:161] P 5a60a9fab026458293b88d08fb10e88a: Beginning new tablet copy session on tablet d0d4cda1deb247d1872658e08dbd6d6b from peer 857b0b25994641599b6e146cec879443 at {username='slave'} at 127.18.13.133:57361: session id = 857b0b25994641599b6e146cec879443-d0d4cda1deb247d1872658e08dbd6d6b
I20240502 07:59:31.824528 523 tablet_copy_source_session.cc:205] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:31.825743 2336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d4cda1deb247d1872658e08dbd6d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:31.827665 2336 tablet_copy_client.cc:783] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:31.827859 2336 tablet_copy_client.cc:647] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:31.829222 2336 tablet_copy_client.cc:523] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:31.830220 2336 tablet_bootstrap.cc:492] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443: Bootstrap starting.
I20240502 07:59:31.847687 2336 tablet_bootstrap.cc:492] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:31.848033 2336 tablet_bootstrap.cc:492] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443: Bootstrap complete.
I20240502 07:59:31.848217 2336 ts_tablet_manager.cc:1367] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.001s
I20240502 07:59:31.848641 2336 raft_consensus.cc:348] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } }
I20240502 07:59:31.848750 2336 raft_consensus.cc:729] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 857b0b25994641599b6e146cec879443, State: Initialized, Role: LEARNER
I20240502 07:59:31.848858 2336 consensus_queue.cc:260] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: true } }
I20240502 07:59:31.849207 2336 ts_tablet_manager.cc:1398] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:31.849948 523 tablet_copy_service.cc:342] P 5a60a9fab026458293b88d08fb10e88a: Request end of tablet copy session 857b0b25994641599b6e146cec879443-d0d4cda1deb247d1872658e08dbd6d6b received from {username='slave'} at 127.18.13.133:57361
I20240502 07:59:31.850040 523 tablet_copy_service.cc:433] P 5a60a9fab026458293b88d08fb10e88a: ending tablet copy session 857b0b25994641599b6e146cec879443-d0d4cda1deb247d1872658e08dbd6d6b on tablet d0d4cda1deb247d1872658e08dbd6d6b with peer 857b0b25994641599b6e146cec879443
I20240502 07:59:32.171387 1516 raft_consensus.cc:1203] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:32.377933 1648 raft_consensus.cc:1203] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:32.559091 1848 raft_consensus.cc:1050] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec: attempting to promote NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 to VOTER
I20240502 07:59:32.559533 1848 consensus_queue.cc:237] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } }
I20240502 07:59:32.561000 371 raft_consensus.cc:1261] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:32.561009 1516 raft_consensus.cc:1261] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:32.561049 501 raft_consensus.cc:1261] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:32.561504 2391 consensus_queue.cc:1035] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:32.561720 2365 consensus_queue.cc:1035] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:32.561909 1848 consensus_queue.cc:1035] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:32.563834 2365 raft_consensus.cc:2932] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:32.564312 1516 raft_consensus.cc:2932] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:32.565052 501 raft_consensus.cc:2932] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:32.566720 32695 catalog_manager.cc:5484] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 reported cstate change: config changed from index 4 to 5, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:32.566985 371 raft_consensus.cc:2932] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:32.571640 634 consensus_queue.cc:237] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } }
I20240502 07:59:32.572872 1516 raft_consensus.cc:1261] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:32.572912 371 raft_consensus.cc:1261] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:32.573230 1848 consensus_queue.cc:1035] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:32.573413 2365 consensus_queue.cc:1035] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:32.574828 2365 raft_consensus.cc:2932] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:32.575502 371 raft_consensus.cc:2932] T 9bf16ec8c5854b2daaccabe49d10ba26 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:32.576263 32683 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9bf16ec8c5854b2daaccabe49d10ba26 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:32.576418 1516 raft_consensus.cc:2932] T 9bf16ec8c5854b2daaccabe49d10ba26 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:32.578055 32688 catalog_manager.cc:5484] T 9bf16ec8c5854b2daaccabe49d10ba26 P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index 5 to 6, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:32.580322 483 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bf16ec8c5854b2daaccabe49d10ba26 with delete_type TABLET_DATA_TOMBSTONED (TS 5a60a9fab026458293b88d08fb10e88a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36396
I20240502 07:59:32.580684 2399 tablet_replica.cc:330] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a: stopping tablet replica
I20240502 07:59:32.580866 2399 raft_consensus.cc:2229] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:32.581090 2399 raft_consensus.cc:2258] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:32.581688 2399 ts_tablet_manager.cc:1875] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:32.583325 2399 ts_tablet_manager.cc:1888] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:32.583388 2399 log.cc:1196] T 9bf16ec8c5854b2daaccabe49d10ba26 P 5a60a9fab026458293b88d08fb10e88a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/9bf16ec8c5854b2daaccabe49d10ba26
I20240502 07:59:32.583902 32681 catalog_manager.cc:4830] TS 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457): tablet 9bf16ec8c5854b2daaccabe49d10ba26 (table rebalance_test_table_1 [id=a353f7ca52eb4a0e86f45854ebf5dbe6]) successfully deleted
I20240502 07:59:32.815544 2378 raft_consensus.cc:1050] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a: attempting to promote NON_VOTER 857b0b25994641599b6e146cec879443 to VOTER
I20240502 07:59:32.816063 2378 consensus_queue.cc:237] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } }
I20240502 07:59:32.817521 1648 raft_consensus.cc:1261] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443 [term 1 LEARNER]: Refusing update from remote peer 5a60a9fab026458293b88d08fb10e88a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:32.817521 634 raft_consensus.cc:1261] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Refusing update from remote peer 5a60a9fab026458293b88d08fb10e88a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:32.817668 371 raft_consensus.cc:1261] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 5a60a9fab026458293b88d08fb10e88a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:32.817970 2378 consensus_queue.cc:1035] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:32.818190 2295 consensus_queue.cc:1035] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:32.818378 2416 consensus_queue.cc:1035] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:32.820227 2295 raft_consensus.cc:2932] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 857b0b25994641599b6e146cec879443 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:32.820726 1648 raft_consensus.cc:2932] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 857b0b25994641599b6e146cec879443 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:32.820963 371 raft_consensus.cc:2932] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 857b0b25994641599b6e146cec879443 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:32.823068 634 raft_consensus.cc:2932] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 857b0b25994641599b6e146cec879443 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:32.823426 32696 catalog_manager.cc:5484] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a reported cstate change: config changed from index 4 to 5, 857b0b25994641599b6e146cec879443 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5a60a9fab026458293b88d08fb10e88a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:32.826251 501 consensus_queue.cc:237] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } }
I20240502 07:59:32.827327 1648 raft_consensus.cc:1261] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443 [term 1 FOLLOWER]: Refusing update from remote peer 5a60a9fab026458293b88d08fb10e88a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:32.827399 371 raft_consensus.cc:1261] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 5a60a9fab026458293b88d08fb10e88a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:32.827759 2377 consensus_queue.cc:1035] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:32.827968 2416 consensus_queue.cc:1035] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [LEADER]: Connected to new peer: Peer: permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:32.829416 2377 raft_consensus.cc:2932] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 733b523e2f3e43539b480b034726d7ec (127.18.13.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:32.830042 371 raft_consensus.cc:2932] T d0d4cda1deb247d1872658e08dbd6d6b P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 733b523e2f3e43539b480b034726d7ec (127.18.13.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:32.830907 32681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d0d4cda1deb247d1872658e08dbd6d6b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:32.831135 1648 raft_consensus.cc:2932] T d0d4cda1deb247d1872658e08dbd6d6b P 857b0b25994641599b6e146cec879443 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 733b523e2f3e43539b480b034726d7ec (127.18.13.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } } }
I20240502 07:59:32.831619 32688 catalog_manager.cc:5484] T d0d4cda1deb247d1872658e08dbd6d6b P 5a60a9fab026458293b88d08fb10e88a reported cstate change: config changed from index 5 to 6, VOTER 733b523e2f3e43539b480b034726d7ec (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "5a60a9fab026458293b88d08fb10e88a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "857b0b25994641599b6e146cec879443" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 44143 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:32.834125 614 tablet_service.cc:1514] Processing DeleteTablet for tablet d0d4cda1deb247d1872658e08dbd6d6b with delete_type TABLET_DATA_TOMBSTONED (TS 733b523e2f3e43539b480b034726d7ec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56412
I20240502 07:59:32.834501 2420 tablet_replica.cc:330] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec: stopping tablet replica
I20240502 07:59:32.834636 2420 raft_consensus.cc:2229] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:32.834831 2420 raft_consensus.cc:2258] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:32.835434 2420 ts_tablet_manager.cc:1875] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:32.837139 2420 ts_tablet_manager.cc:1888] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:32.837224 2420 log.cc:1196] T d0d4cda1deb247d1872658e08dbd6d6b P 733b523e2f3e43539b480b034726d7ec: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/d0d4cda1deb247d1872658e08dbd6d6b
I20240502 07:59:32.837711 32683 catalog_manager.cc:4830] TS 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189): tablet d0d4cda1deb247d1872658e08dbd6d6b (table rebalance_test_table_2 [id=63eb7086af244504b9daf9beaeb2a9bd]) successfully deleted
I20240502 07:59:32.987291 614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:32.988368 1367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:32.989967 1498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:32.990250 349 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:32.994583 1629 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:33.010012 483 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:33.089885 614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:33.091265 483 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:33.092334 349 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:33.094936 1498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:33.106226 1367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:33.110931 1629 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:33.142521 635 consensus_queue.cc:237] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } }
I20240502 07:59:33.144069 1516 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 07:59:33.144111 568 consensus_peers.cc:485] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec -> Peer 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132:33885): Couldn't send request to peer 468fd6f1f0fb46ce87ab94f922e89661. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e199f4ff646b4992ab7c6d5527ac32db. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:33.144505 371 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:33.144548 2512 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:33.144918 1848 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:33.146486 2512 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } }
I20240502 07:59:33.146932 1516 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } }
I20240502 07:59:33.148101 371 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } }
I20240502 07:59:33.150434 32692 catalog_manager.cc:5484] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 reported cstate change: config changed from index 6 to 7, NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } }
I20240502 07:59:33.158761 634 consensus_queue.cc:237] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false replace: true } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } }
I20240502 07:59:33.160593 501 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 07:59:33.160645 569 consensus_peers.cc:485] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec -> Peer 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134:39939): Couldn't send request to peer 783ed3162fe14b76b9cb7a7a6b8bdaa9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42d23c677d594a6693a052ca2adc2de8. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:33.160629 1387 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:33.161039 1848 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:33.161233 2512 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:33.163049 2391 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false replace: true } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } } }
I20240502 07:59:33.165084 1387 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false replace: true } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } } }
I20240502 07:59:33.165248 501 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false replace: true } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } } }
I20240502 07:59:33.167750 32688 catalog_manager.cc:5484] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index 6 to 7, NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) added. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:33.202106 349 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:33.203824 1367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:33.204190 1498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:33.205781 483 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:33.215301 614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:33.219355 1629 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:33.253659 634 consensus_queue.cc:237] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: true } }
I20240502 07:59:33.255131 1387 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:33.255151 501 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:33.255574 1848 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:33.255825 2391 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 07:59:33.256065 568 consensus_peers.cc:485] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec -> Peer 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429): Couldn't send request to peer 2ca73afa63a04c378a0948e9691601fd. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:33.257699 2512 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: true } } }
I20240502 07:59:33.258180 501 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: true } } }
I20240502 07:59:33.260058 1387 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: true } } }
I20240502 07:59:33.260586 32692 catalog_manager.cc:5484] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a reported cstate change: config changed from index 6 to 7, NON_VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) added. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: true } } }
I20240502 07:59:33.268714 634 consensus_queue.cc:237] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: true } }
I20240502 07:59:33.270303 1516 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 07:59:33.270476 568 consensus_peers.cc:485] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec -> Peer 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132:33885): Couldn't send request to peer 468fd6f1f0fb46ce87ab94f922e89661. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e199f4ff646b4992ab7c6d5527ac32db. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:33.270509 371 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:33.270701 2512 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 07:59:33.271036 568 consensus_peers.cc:485] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec -> Peer 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457): Couldn't send request to peer 5a60a9fab026458293b88d08fb10e88a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:33.271209 2512 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:33.272387 2391 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: true } } }
I20240502 07:59:33.273197 371 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: true } } }
I20240502 07:59:33.274174 1516 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: true } } }
I20240502 07:59:33.275655 32696 catalog_manager.cc:5484] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index 7 to 8, NON_VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) added. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:33.699757 2618 ts_tablet_manager.cc:902] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: Initiating tablet copy from peer 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:33.700575 2618 tablet_copy_client.cc:308] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:37189
I20240502 07:59:33.701192 655 tablet_copy_service.cc:140] P 733b523e2f3e43539b480b034726d7ec: Received BeginTabletCopySession request for tablet e199f4ff646b4992ab7c6d5527ac32db from peer 5a60a9fab026458293b88d08fb10e88a ({username='slave'} at 127.18.13.130:42883)
I20240502 07:59:33.701319 655 tablet_copy_service.cc:161] P 733b523e2f3e43539b480b034726d7ec: Beginning new tablet copy session on tablet e199f4ff646b4992ab7c6d5527ac32db from peer 5a60a9fab026458293b88d08fb10e88a at {username='slave'} at 127.18.13.130:42883: session id = 5a60a9fab026458293b88d08fb10e88a-e199f4ff646b4992ab7c6d5527ac32db
I20240502 07:59:33.702253 655 tablet_copy_source_session.cc:205] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:33.703333 2618 ts_tablet_manager.cc:1875] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 07:59:33.705308 2618 ts_tablet_manager.cc:1888] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 07:59:33.705504 2618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e199f4ff646b4992ab7c6d5527ac32db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:33.706753 2618 tablet_copy_client.cc:783] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:33.706954 2618 tablet_copy_client.cc:647] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:33.708343 2618 tablet_copy_client.cc:523] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:33.709928 2618 tablet_bootstrap.cc:492] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: Bootstrap starting.
I20240502 07:59:33.714673 2634 ts_tablet_manager.cc:902] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Initiating tablet copy from peer 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:33.715296 2634 tablet_copy_client.cc:308] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:37189
I20240502 07:59:33.715739 655 tablet_copy_service.cc:140] P 733b523e2f3e43539b480b034726d7ec: Received BeginTabletCopySession request for tablet 42d23c677d594a6693a052ca2adc2de8 from peer 783ed3162fe14b76b9cb7a7a6b8bdaa9 ({username='slave'} at 127.18.13.134:41773)
I20240502 07:59:33.715868 655 tablet_copy_service.cc:161] P 733b523e2f3e43539b480b034726d7ec: Beginning new tablet copy session on tablet 42d23c677d594a6693a052ca2adc2de8 from peer 783ed3162fe14b76b9cb7a7a6b8bdaa9 at {username='slave'} at 127.18.13.134:41773: session id = 783ed3162fe14b76b9cb7a7a6b8bdaa9-42d23c677d594a6693a052ca2adc2de8
I20240502 07:59:33.716840 655 tablet_copy_source_session.cc:205] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:33.719023 2634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d23c677d594a6693a052ca2adc2de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:33.721540 2634 tablet_copy_client.cc:783] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:33.721688 2634 tablet_copy_client.cc:647] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:33.722837 2634 tablet_copy_client.cc:523] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:33.723881 2634 tablet_bootstrap.cc:492] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Bootstrap starting.
I20240502 07:59:33.732980 2618 tablet_bootstrap.cc:492] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:33.733333 2618 tablet_bootstrap.cc:492] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: Bootstrap complete.
I20240502 07:59:33.733536 2618 ts_tablet_manager.cc:1367] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: Time spent bootstrapping tablet: real 0.024s user 0.022s sys 0.001s
I20240502 07:59:33.734030 2618 raft_consensus.cc:348] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: true } }
I20240502 07:59:33.734156 2618 raft_consensus.cc:729] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a60a9fab026458293b88d08fb10e88a, State: Initialized, Role: LEARNER
I20240502 07:59:33.734268 2618 consensus_queue.cc:260] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: true } }
I20240502 07:59:33.734606 2618 ts_tablet_manager.cc:1398] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 07:59:33.735293 655 tablet_copy_service.cc:342] P 733b523e2f3e43539b480b034726d7ec: Request end of tablet copy session 5a60a9fab026458293b88d08fb10e88a-e199f4ff646b4992ab7c6d5527ac32db received from {username='slave'} at 127.18.13.130:42883
I20240502 07:59:33.735388 655 tablet_copy_service.cc:433] P 733b523e2f3e43539b480b034726d7ec: ending tablet copy session 5a60a9fab026458293b88d08fb10e88a-e199f4ff646b4992ab7c6d5527ac32db on tablet e199f4ff646b4992ab7c6d5527ac32db with peer 5a60a9fab026458293b88d08fb10e88a
I20240502 07:59:33.744498 2634 tablet_bootstrap.cc:492] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:33.744800 2634 tablet_bootstrap.cc:492] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Bootstrap complete.
I20240502 07:59:33.744972 2634 ts_tablet_manager.cc:1367] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.003s
I20240502 07:59:33.745416 2634 raft_consensus.cc:348] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false replace: true } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } }
I20240502 07:59:33.745527 2634 raft_consensus.cc:729] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 783ed3162fe14b76b9cb7a7a6b8bdaa9, State: Initialized, Role: LEARNER
I20240502 07:59:33.745631 2634 consensus_queue.cc:260] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false replace: true } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: NON_VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: true } }
I20240502 07:59:33.745878 2634 ts_tablet_manager.cc:1398] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:33.746465 655 tablet_copy_service.cc:342] P 733b523e2f3e43539b480b034726d7ec: Request end of tablet copy session 783ed3162fe14b76b9cb7a7a6b8bdaa9-42d23c677d594a6693a052ca2adc2de8 received from {username='slave'} at 127.18.13.134:41773
I20240502 07:59:33.746564 655 tablet_copy_service.cc:433] P 733b523e2f3e43539b480b034726d7ec: ending tablet copy session 783ed3162fe14b76b9cb7a7a6b8bdaa9-42d23c677d594a6693a052ca2adc2de8 on tablet 42d23c677d594a6693a052ca2adc2de8 with peer 783ed3162fe14b76b9cb7a7a6b8bdaa9
I20240502 07:59:33.755162 2640 ts_tablet_manager.cc:902] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661: Initiating tablet copy from peer 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:33.755681 2640 tablet_copy_client.cc:308] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:37189
I20240502 07:59:33.756214 655 tablet_copy_service.cc:140] P 733b523e2f3e43539b480b034726d7ec: Received BeginTabletCopySession request for tablet e199f4ff646b4992ab7c6d5527ac32db from peer 468fd6f1f0fb46ce87ab94f922e89661 ({username='slave'} at 127.18.13.132:33499)
I20240502 07:59:33.756319 655 tablet_copy_service.cc:161] P 733b523e2f3e43539b480b034726d7ec: Beginning new tablet copy session on tablet e199f4ff646b4992ab7c6d5527ac32db from peer 468fd6f1f0fb46ce87ab94f922e89661 at {username='slave'} at 127.18.13.132:33499: session id = 468fd6f1f0fb46ce87ab94f922e89661-e199f4ff646b4992ab7c6d5527ac32db
I20240502 07:59:33.756992 655 tablet_copy_source_session.cc:205] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:33.757833 2640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e199f4ff646b4992ab7c6d5527ac32db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:33.759575 2640 tablet_copy_client.cc:783] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:33.759728 2640 tablet_copy_client.cc:647] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:33.760830 2640 tablet_copy_client.cc:523] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:33.761953 2640 tablet_bootstrap.cc:492] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661: Bootstrap starting.
I20240502 07:59:33.784431 2640 tablet_bootstrap.cc:492] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:33.784780 2640 tablet_bootstrap.cc:492] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661: Bootstrap complete.
I20240502 07:59:33.784977 2640 ts_tablet_manager.cc:1367] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661: Time spent bootstrapping tablet: real 0.023s user 0.021s sys 0.002s
I20240502 07:59:33.785475 2640 raft_consensus.cc:348] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: true } }
I20240502 07:59:33.785629 2640 raft_consensus.cc:729] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 468fd6f1f0fb46ce87ab94f922e89661, State: Initialized, Role: LEARNER
I20240502 07:59:33.785732 2640 consensus_queue.cc:260] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: true } }
I20240502 07:59:33.786134 2640 ts_tablet_manager.cc:1398] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:33.786872 655 tablet_copy_service.cc:342] P 733b523e2f3e43539b480b034726d7ec: Request end of tablet copy session 468fd6f1f0fb46ce87ab94f922e89661-e199f4ff646b4992ab7c6d5527ac32db received from {username='slave'} at 127.18.13.132:33499
I20240502 07:59:33.786967 655 tablet_copy_service.cc:433] P 733b523e2f3e43539b480b034726d7ec: ending tablet copy session 468fd6f1f0fb46ce87ab94f922e89661-e199f4ff646b4992ab7c6d5527ac32db on tablet e199f4ff646b4992ab7c6d5527ac32db with peer 468fd6f1f0fb46ce87ab94f922e89661
I20240502 07:59:33.841720 2657 ts_tablet_manager.cc:902] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: Initiating tablet copy from peer 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189)
I20240502 07:59:33.842430 2657 tablet_copy_client.cc:308] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:37189
I20240502 07:59:33.843111 655 tablet_copy_service.cc:140] P 733b523e2f3e43539b480b034726d7ec: Received BeginTabletCopySession request for tablet cc751cb753a840d28fb8787ca1ecafb0 from peer 2ca73afa63a04c378a0948e9691601fd ({username='slave'} at 127.18.13.129:48179)
I20240502 07:59:33.843225 655 tablet_copy_service.cc:161] P 733b523e2f3e43539b480b034726d7ec: Beginning new tablet copy session on tablet cc751cb753a840d28fb8787ca1ecafb0 from peer 2ca73afa63a04c378a0948e9691601fd at {username='slave'} at 127.18.13.129:48179: session id = 2ca73afa63a04c378a0948e9691601fd-cc751cb753a840d28fb8787ca1ecafb0
I20240502 07:59:33.844020 655 tablet_copy_source_session.cc:205] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:33.845023 2657 ts_tablet_manager.cc:1875] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 07:59:33.846925 2657 ts_tablet_manager.cc:1888] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 07:59:33.847103 2657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc751cb753a840d28fb8787ca1ecafb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:33.848234 2657 tablet_copy_client.cc:783] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:33.848416 2657 tablet_copy_client.cc:647] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:33.849886 2657 tablet_copy_client.cc:523] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:33.851246 2657 tablet_bootstrap.cc:492] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: Bootstrap starting.
I20240502 07:59:33.870469 2657 tablet_bootstrap.cc:492] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:33.870831 2657 tablet_bootstrap.cc:492] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: Bootstrap complete.
I20240502 07:59:33.871022 2657 ts_tablet_manager.cc:1367] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.003s
I20240502 07:59:33.871454 2657 raft_consensus.cc:348] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: true } }
I20240502 07:59:33.871572 2657 raft_consensus.cc:729] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca73afa63a04c378a0948e9691601fd, State: Initialized, Role: LEARNER
I20240502 07:59:33.871703 2657 consensus_queue.cc:260] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: NON_VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: true } }
I20240502 07:59:33.871990 2657 ts_tablet_manager.cc:1398] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 07:59:33.873420 655 tablet_copy_service.cc:342] P 733b523e2f3e43539b480b034726d7ec: Request end of tablet copy session 2ca73afa63a04c378a0948e9691601fd-cc751cb753a840d28fb8787ca1ecafb0 received from {username='slave'} at 127.18.13.129:48179
I20240502 07:59:33.873525 655 tablet_copy_service.cc:433] P 733b523e2f3e43539b480b034726d7ec: ending tablet copy session 2ca73afa63a04c378a0948e9691601fd-cc751cb753a840d28fb8787ca1ecafb0 on tablet cc751cb753a840d28fb8787ca1ecafb0 with peer 2ca73afa63a04c378a0948e9691601fd
I20240502 07:59:34.162096 1387 raft_consensus.cc:1203] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 07:59:34.204485 1516 raft_consensus.cc:1203] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 07:59:34.216236 501 raft_consensus.cc:1203] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 07:59:34.407145 371 raft_consensus.cc:1203] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 07:59:34.706254 1848 raft_consensus.cc:1050] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec: attempting to promote NON_VOTER 5a60a9fab026458293b88d08fb10e88a to VOTER
I20240502 07:59:34.706684 1848 consensus_queue.cc:237] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } }
I20240502 07:59:34.708173 1516 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:34.708218 501 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 LEARNER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:34.708292 1387 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:34.708323 371 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:34.708629 2577 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:34.708788 2693 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:34.708951 2746 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:34.709091 2745 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:34.711421 2577 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.711891 371 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.713348 501 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.714421 32692 catalog_manager.cc:5484] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd reported cstate change: config changed from index 8 to 9, 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.714989 1387 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.716135 2745 raft_consensus.cc:1050] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec: attempting to promote NON_VOTER 468fd6f1f0fb46ce87ab94f922e89661 to VOTER
I20240502 07:59:34.716521 2745 consensus_queue.cc:237] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } }
I20240502 07:59:34.716598 1516 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: true } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.718482 501 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:34.718575 371 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:34.718772 1387 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 LEARNER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:34.718971 2693 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:34.719220 2745 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:34.720224 2745 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:34.720986 2746 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.721400 1387 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
W20240502 07:59:34.723551 32683 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet e199f4ff646b4992ab7c6d5527ac32db with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER failed with leader 733b523e2f3e43539b480b034726d7ec (127.18.13.131:37189) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 9 but the committed config has opid_index of 10
I20240502 07:59:34.723521 371 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.725669 32688 catalog_manager.cc:5484] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661 reported cstate change: config changed from index 9 to 10, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.726688 501 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.726944 1516 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:34.727346 2577 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:34.728437 1516 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false replace: true } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.731727 634 consensus_queue.cc:237] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } }
I20240502 07:59:34.733701 1387 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:59:34.733794 501 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:59:34.733837 371 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:59:34.734133 1848 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:34.734263 2745 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:34.734338 2577 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:34.736065 2577 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.736475 371 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.736599 1387 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.737661 32683 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e199f4ff646b4992ab7c6d5527ac32db with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:34.738299 501 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.738729 32692 catalog_manager.cc:5484] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index 10 to 11, VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) evicted. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:34.742285 634 consensus_queue.cc:237] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } }
I20240502 07:59:34.743417 501 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 07:59:34.743572 1387 raft_consensus.cc:1261] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 07:59:34.743772 2745 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:59:34.744115 2745 consensus_queue.cc:1035] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:59:34.745748 1848 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.746963 32683 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e199f4ff646b4992ab7c6d5527ac32db with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:34.746953 1498 tablet_service.cc:1514] Processing DeleteTablet for tablet e199f4ff646b4992ab7c6d5527ac32db with delete_type TABLET_DATA_TOMBSTONED (TS 783ed3162fe14b76b9cb7a7a6b8bdaa9 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:53428
I20240502 07:59:34.747383 1387 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.747499 2760 tablet_replica.cc:330] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9: stopping tablet replica
I20240502 07:59:34.747681 32695 catalog_manager.cc:5484] T e199f4ff646b4992ab7c6d5527ac32db P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index 11 to 12, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:34.747943 2760 raft_consensus.cc:2229] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:34.748173 2760 raft_consensus.cc:2258] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:34.749085 501 raft_consensus.cc:2932] T e199f4ff646b4992ab7c6d5527ac32db P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { promote: false } } }
I20240502 07:59:34.749402 2760 ts_tablet_manager.cc:1875] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:34.750370 349 tablet_service.cc:1514] Processing DeleteTablet for tablet e199f4ff646b4992ab7c6d5527ac32db with delete_type TABLET_DATA_TOMBSTONED (TS 2ca73afa63a04c378a0948e9691601fd not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:54562
I20240502 07:59:34.750710 2761 tablet_replica.cc:330] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd: stopping tablet replica
I20240502 07:59:34.750849 2761 raft_consensus.cc:2229] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:34.751428 2761 raft_consensus.cc:2258] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:34.752386 2761 ts_tablet_manager.cc:1875] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:34.752779 2760 ts_tablet_manager.cc:1888] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 07:59:34.752864 2760 log.cc:1196] T e199f4ff646b4992ab7c6d5527ac32db P 783ed3162fe14b76b9cb7a7a6b8bdaa9: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-5/wal/wals/e199f4ff646b4992ab7c6d5527ac32db
I20240502 07:59:34.753420 32682 catalog_manager.cc:4830] TS 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134:39939): tablet e199f4ff646b4992ab7c6d5527ac32db (table rebalance_test_table_2 [id=63eb7086af244504b9daf9beaeb2a9bd]) successfully deleted
I20240502 07:59:34.754480 2761 ts_tablet_manager.cc:1888] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 07:59:34.754551 2761 log.cc:1196] T e199f4ff646b4992ab7c6d5527ac32db P 2ca73afa63a04c378a0948e9691601fd: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/e199f4ff646b4992ab7c6d5527ac32db
I20240502 07:59:34.755002 32681 catalog_manager.cc:4830] TS 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429): tablet e199f4ff646b4992ab7c6d5527ac32db (table rebalance_test_table_2 [id=63eb7086af244504b9daf9beaeb2a9bd]) successfully deleted
I20240502 07:59:34.794692 2577 raft_consensus.cc:1050] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec: attempting to promote NON_VOTER 783ed3162fe14b76b9cb7a7a6b8bdaa9 to VOTER
I20240502 07:59:34.795264 2577 consensus_queue.cc:237] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false replace: true } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } }
I20240502 07:59:34.796803 1516 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 LEARNER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:34.796828 1387 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:34.796875 501 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:34.797236 1848 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:34.797443 2693 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:34.797700 2693 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:34.800297 1848 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false replace: true } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:34.800745 1516 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false replace: true } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:34.802467 501 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false replace: true } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:34.802873 1387 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false replace: true } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:34.802913 32695 catalog_manager.cc:5484] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index 7 to 8, 783ed3162fe14b76b9cb7a7a6b8bdaa9 (127.18.13.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:34.806744 634 consensus_queue.cc:237] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } }
I20240502 07:59:34.807994 1516 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:34.808094 501 raft_consensus.cc:1261] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:34.808378 2577 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:34.808519 1848 consensus_queue.cc:1035] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:34.809859 1848 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:34.810293 1516 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 783ed3162fe14b76b9cb7a7a6b8bdaa9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:34.810412 501 raft_consensus.cc:2932] T 42d23c677d594a6693a052ca2adc2de8 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } } }
I20240502 07:59:34.811612 32683 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 42d23c677d594a6693a052ca2adc2de8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:34.812057 32696 catalog_manager.cc:5484] T 42d23c677d594a6693a052ca2adc2de8 P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index 8 to 9, VOTER 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132) evicted. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "783ed3162fe14b76b9cb7a7a6b8bdaa9" member_type: VOTER last_known_addr { host: "127.18.13.134" port: 39939 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:34.817703 1367 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d23c677d594a6693a052ca2adc2de8 with delete_type TABLET_DATA_TOMBSTONED (TS 468fd6f1f0fb46ce87ab94f922e89661 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:45620
I20240502 07:59:34.818141 2782 tablet_replica.cc:330] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661: stopping tablet replica
I20240502 07:59:34.818435 2782 raft_consensus.cc:2229] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:34.818683 2782 raft_consensus.cc:2258] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:34.819545 2782 ts_tablet_manager.cc:1875] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:34.821527 2782 ts_tablet_manager.cc:1888] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 07:59:34.821611 2782 log.cc:1196] T 42d23c677d594a6693a052ca2adc2de8 P 468fd6f1f0fb46ce87ab94f922e89661: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/wals/42d23c677d594a6693a052ca2adc2de8
I20240502 07:59:34.822085 32681 catalog_manager.cc:4830] TS 468fd6f1f0fb46ce87ab94f922e89661 (127.18.13.132:33885): tablet 42d23c677d594a6693a052ca2adc2de8 (table rebalance_test_table_0 [id=442b95cfe4e84e99965954832dd504ff]) successfully deleted
I20240502 07:59:34.926692 614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:34.929049 349 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:34.930315 1367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:34.931035 1498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:34.945756 1629 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:34.964533 483 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:35.019721 2577 raft_consensus.cc:1050] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec: attempting to promote NON_VOTER 2ca73afa63a04c378a0948e9691601fd to VOTER
I20240502 07:59:35.020150 2577 consensus_queue.cc:237] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: false } }
I20240502 07:59:35.021580 1387 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:35.021636 501 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:35.021689 371 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 1 LEARNER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:35.022063 2693 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:35.022210 2577 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:35.022359 1848 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:35.024258 2577 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: false } } }
I20240502 07:59:35.024820 1387 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: false } } }
I20240502 07:59:35.024816 501 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: false } } }
I20240502 07:59:35.027072 32695 catalog_manager.cc:5484] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index 7 to 8, 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:35.027096 371 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2ca73afa63a04c378a0948e9691601fd (127.18.13.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5a60a9fab026458293b88d08fb10e88a" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 37457 } attrs { replace: true } } peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: false } } }
I20240502 07:59:35.030241 634 consensus_queue.cc:237] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: false } }
I20240502 07:59:35.031276 371 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:35.031324 1387 raft_consensus.cc:1261] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Refusing update from remote peer 733b523e2f3e43539b480b034726d7ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:35.031643 2693 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:35.031847 2746 consensus_queue.cc:1035] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:35.033325 2746 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: false } } }
I20240502 07:59:35.033764 371 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 2ca73afa63a04c378a0948e9691601fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: false } } }
I20240502 07:59:35.034266 1387 raft_consensus.cc:2932] T cc751cb753a840d28fb8787ca1ecafb0 P 468fd6f1f0fb46ce87ab94f922e89661 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: false } } }
I20240502 07:59:35.034852 32683 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cc751cb753a840d28fb8787ca1ecafb0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:35.035758 32690 catalog_manager.cc:5484] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec reported cstate change: config changed from index 8 to 9, VOTER 5a60a9fab026458293b88d08fb10e88a (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "733b523e2f3e43539b480b034726d7ec" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "733b523e2f3e43539b480b034726d7ec" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 37189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "468fd6f1f0fb46ce87ab94f922e89661" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 33885 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca73afa63a04c378a0948e9691601fd" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 46429 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:35.038184 483 tablet_service.cc:1514] Processing DeleteTablet for tablet cc751cb753a840d28fb8787ca1ecafb0 with delete_type TABLET_DATA_TOMBSTONED (TS 5a60a9fab026458293b88d08fb10e88a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36396
I20240502 07:59:35.038569 2834 tablet_replica.cc:330] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a: stopping tablet replica
I20240502 07:59:35.038759 2834 raft_consensus.cc:2229] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:35.038959 2834 raft_consensus.cc:2258] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:35.039554 2834 ts_tablet_manager.cc:1875] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:35.041445 2834 ts_tablet_manager.cc:1888] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 07:59:35.041517 2834 log.cc:1196] T cc751cb753a840d28fb8787ca1ecafb0 P 5a60a9fab026458293b88d08fb10e88a: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/cc751cb753a840d28fb8787ca1ecafb0
I20240502 07:59:35.042021 32681 catalog_manager.cc:4830] TS 5a60a9fab026458293b88d08fb10e88a (127.18.13.130:37457): tablet cc751cb753a840d28fb8787ca1ecafb0 (table rebalance_test_table_0 [id=442b95cfe4e84e99965954832dd504ff]) successfully deleted
I20240502 07:59:35.262709 349 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:35.264539 614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:35.268580 483 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:35.270052 1498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:35.277858 1367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:35.288272 1629 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:35.361595 1367 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:35.362449 349 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:35.366267 614 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:35.377502 1498 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:35.386651 483 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:35.388994 1629 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:35.419320 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 32757
I20240502 07:59:35.426290 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 420
W20240502 07:59:35.428594 568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.13.129:46429: connect: Connection refused (error 111)
W20240502 07:59:35.429446 568 consensus_peers.cc:485] T cc751cb753a840d28fb8787ca1ecafb0 P 733b523e2f3e43539b480b034726d7ec -> Peer 2ca73afa63a04c378a0948e9691601fd (127.18.13.129:46429): Couldn't send request to peer 2ca73afa63a04c378a0948e9691601fd. Status: Network error: Client connection negotiation failed: client connection to 127.18.13.129:46429: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:59:35.432355 568 connection.cc:535] client connection to 127.18.13.130:37457 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 07:59:35.432849 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 552
I20240502 07:59:35.439564 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 1305
I20240502 07:59:35.445192 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 1567
I20240502 07:59:35.450804 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 1436
I20240502 07:59:35.458367 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 32666
2024-05-02T07:59:35Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3 (14139 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2
I20240502 07:59:35.487190 18486 test_util.cc:268] Using random seed: -68421937
I20240502 07:59:35.488056 18486 ts_itest-base.cc:112] Starting cluster with:
I20240502 07:59:35.488096 18486 ts_itest-base.cc:113] --------------
I20240502 07:59:35.488116 18486 ts_itest-base.cc:114] 5 tablet servers
I20240502 07:59:35.488132 18486 ts_itest-base.cc:115] 3 replicas per TS
I20240502 07:59:35.488148 18486 ts_itest-base.cc:116] --------------
2024-05-02T07:59:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T07:59:35Z Disabled control of system clock
I20240502 07:59:35.495983 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.13.190:45049
--webserver_interface=127.18.13.190
--webserver_port=0
--builtin_ntp_servers=127.18.13.148:37855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.13.190:45049
--location_mapping_cmd=/tmp/dist-test-taskeLuhys/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 07:59:35.602286 2905 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 07:59:35.602555 2905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:35.602612 2905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:35.602689 2905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:35.605676 2905 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:35.605767 2905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 07:59:35.605818 2905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:35.605851 2905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 07:59:35.605886 2905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 07:59:35.609472 2905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:37855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskeLuhys/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.18.13.190:45049
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.13.190:45049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.18.13.190
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:35.610419 2905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:35.611550 2905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:35.619318 2910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:35.619925 2905 server_base.cc:1034] running on GCE node
W20240502 07:59:35.620546 2913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:35.621137 2911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:35.621623 2905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:35.622532 2905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:35.623685 2905 hybrid_clock.cc:648] HybridClock initialized: now 1714636775623674 us; error 27 us; skew 500 ppm
I20240502 07:59:35.625533 2905 webserver.cc:460] Webserver started at http://127.18.13.190:34453/ using document root <none> and password file <none>
I20240502 07:59:35.626096 2905 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:35.626159 2905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:35.626328 2905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:35.628010 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "13794e7bf30241459e40a4fa8ba38e46"
format_stamp: "Formatted at 2024-05-02 07:59:35 on dist-test-slave-f46z"
I20240502 07:59:35.628494 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "13794e7bf30241459e40a4fa8ba38e46"
format_stamp: "Formatted at 2024-05-02 07:59:35 on dist-test-slave-f46z"
I20240502 07:59:35.631628 2905 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 07:59:35.633766 2919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:35.634694 2905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:35.634824 2905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
uuid: "13794e7bf30241459e40a4fa8ba38e46"
format_stamp: "Formatted at 2024-05-02 07:59:35 on dist-test-slave-f46z"
I20240502 07:59:35.634903 2905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:35.653990 2905 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:35.654670 2905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:35.654843 2905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:35.661628 2905 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.190:45049
I20240502 07:59:35.661635 2971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.190:45049 every 8 connection(s)
I20240502 07:59:35.662607 2905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 07:59:35.665133 2972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:35.669945 2972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46: Bootstrap starting.
I20240502 07:59:35.671064 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 2905
I20240502 07:59:35.671198 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 07:59:35.672467 2972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:35.673187 2972 log.cc:824] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:35.674820 2972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46: No bootstrap required, opened a new log
I20240502 07:59:35.677330 2972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13794e7bf30241459e40a4fa8ba38e46" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 45049 } }
I20240502 07:59:35.677527 2972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:35.677604 2972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13794e7bf30241459e40a4fa8ba38e46, State: Initialized, Role: FOLLOWER
I20240502 07:59:35.678015 2972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13794e7bf30241459e40a4fa8ba38e46" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 45049 } }
I20240502 07:59:35.678141 2972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:35.678201 2972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:35.678320 2972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:35.679199 2972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13794e7bf30241459e40a4fa8ba38e46" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 45049 } }
I20240502 07:59:35.679510 2972 leader_election.cc:302] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13794e7bf30241459e40a4fa8ba38e46; no voters:
I20240502 07:59:35.679759 2972 leader_election.cc:288] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:35.679934 2977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:35.680166 2977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [term 1 LEADER]: Becoming Leader. State: Replica: 13794e7bf30241459e40a4fa8ba38e46, State: Running, Role: LEADER
I20240502 07:59:35.680435 2977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13794e7bf30241459e40a4fa8ba38e46" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 45049 } }
I20240502 07:59:35.680868 2972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:59:35.681491 2977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13794e7bf30241459e40a4fa8ba38e46. Latest consensus state: current_term: 1 leader_uuid: "13794e7bf30241459e40a4fa8ba38e46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13794e7bf30241459e40a4fa8ba38e46" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 45049 } } }
I20240502 07:59:35.681617 2977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:35.681910 2978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13794e7bf30241459e40a4fa8ba38e46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13794e7bf30241459e40a4fa8ba38e46" member_type: VOTER last_known_addr { host: "127.18.13.190" port: 45049 } } }
I20240502 07:59:35.682013 2978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46 [sys.catalog]: This master's current role is: LEADER
W20240502 07:59:35.685397 2992 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 07:59:35.685472 2992 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 07:59:35.685546 2983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:59:35.686252 2983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:59:35.691208 2983 catalog_manager.cc:1347] Generated new cluster ID: 36961d942b5642cc86122fb0da4587d9
I20240502 07:59:35.691283 2983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:59:35.707409 2983 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:59:35.708664 2983 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:59:35.717366 2983 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 13794e7bf30241459e40a4fa8ba38e46: Generated new TSK 0
I20240502 07:59:35.718183 2983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:59:35.720808 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.129:0
--local_ip_for_outbound_sockets=127.18.13.129
--webserver_interface=127.18.13.129
--webserver_port=0
--tserver_master_addrs=127.18.13.190:45049
--builtin_ntp_servers=127.18.13.148:37855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 07:59:35.827446 2996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:35.827658 2996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:35.827702 2996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:35.830703 2996 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:35.830763 2996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:35.830835 2996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.129
I20240502 07:59:35.834491 2996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:37855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.18.13.129
--webserver_port=0
--tserver_master_addrs=127.18.13.190:45049
--never_fsync=true
--heap_profile_path=/tmp/kudu.2996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.129
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:35.835341 2996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:35.836496 2996 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:35.844635 3002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:35.845312 3004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:35.845849 3001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:35.846539 2996 server_base.cc:1034] running on GCE node
I20240502 07:59:35.846872 2996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:35.847386 2996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:35.848531 2996 hybrid_clock.cc:648] HybridClock initialized: now 1714636775848513 us; error 32 us; skew 500 ppm
I20240502 07:59:35.850414 2996 webserver.cc:460] Webserver started at http://127.18.13.129:42573/ using document root <none> and password file <none>
I20240502 07:59:35.850944 2996 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:35.850996 2996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:35.851161 2996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:35.852816 2996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d11d320786dd4f1094ce083e43bfd5b5"
format_stamp: "Formatted at 2024-05-02 07:59:35 on dist-test-slave-f46z"
I20240502 07:59:35.853245 2996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d11d320786dd4f1094ce083e43bfd5b5"
format_stamp: "Formatted at 2024-05-02 07:59:35 on dist-test-slave-f46z"
I20240502 07:59:35.856441 2996 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 07:59:35.858435 3010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:35.859396 2996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:35.859498 2996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d11d320786dd4f1094ce083e43bfd5b5"
format_stamp: "Formatted at 2024-05-02 07:59:35 on dist-test-slave-f46z"
I20240502 07:59:35.859591 2996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:35.879920 2996 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:35.880584 2996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:35.880741 2996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:35.881268 2996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:35.882220 2996 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:35.882283 2996 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:35.882342 2996 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:35.882373 2996 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:35.890815 2996 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.129:37063
I20240502 07:59:35.890838 3123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.129:37063 every 8 connection(s)
I20240502 07:59:35.891726 2996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 07:59:35.896701 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 2996
I20240502 07:59:35.896828 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 07:59:35.899078 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.130:0
--local_ip_for_outbound_sockets=127.18.13.130
--webserver_interface=127.18.13.130
--webserver_port=0
--tserver_master_addrs=127.18.13.190:45049
--builtin_ntp_servers=127.18.13.148:37855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:35.900713 3124 heartbeater.cc:347] Connected to a master server at 127.18.13.190:45049
I20240502 07:59:35.900998 3124 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:35.901520 3124 heartbeater.cc:511] Master 127.18.13.190:45049 requested a full tablet report, sending...
I20240502 07:59:35.951243 2936 ts_manager.cc:194] Registered new tserver with Master: d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063)
I20240502 07:59:35.952607 2936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.129:43365
W20240502 07:59:36.020481 3127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:36.020728 3127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:36.020772 3127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:36.023746 3127 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:36.023861 3127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:36.023934 3127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.130
I20240502 07:59:36.027426 3127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:37855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.18.13.130
--webserver_port=0
--tserver_master_addrs=127.18.13.190:45049
--never_fsync=true
--heap_profile_path=/tmp/kudu.3127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.130
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:36.028357 3127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:36.029469 3127 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:36.038074 3134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:36.038573 3136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:36.039135 3133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:36.039932 3127 server_base.cc:1034] running on GCE node
I20240502 07:59:36.040340 3127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:36.040892 3127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:36.042057 3127 hybrid_clock.cc:648] HybridClock initialized: now 1714636776042044 us; error 29 us; skew 500 ppm
I20240502 07:59:36.043999 3127 webserver.cc:460] Webserver started at http://127.18.13.130:41303/ using document root <none> and password file <none>
I20240502 07:59:36.044548 3127 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:36.044610 3127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:36.044809 3127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:36.046430 3127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3dc087b5db164c89bea2988820a9b31c"
format_stamp: "Formatted at 2024-05-02 07:59:36 on dist-test-slave-f46z"
I20240502 07:59:36.046911 3127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3dc087b5db164c89bea2988820a9b31c"
format_stamp: "Formatted at 2024-05-02 07:59:36 on dist-test-slave-f46z"
I20240502 07:59:36.050156 3127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20240502 07:59:36.052284 3142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:36.053221 3127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240502 07:59:36.053330 3127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3dc087b5db164c89bea2988820a9b31c"
format_stamp: "Formatted at 2024-05-02 07:59:36 on dist-test-slave-f46z"
I20240502 07:59:36.053409 3127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:36.071096 3127 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:36.071722 3127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:36.071928 3127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:36.072453 3127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:36.073328 3127 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:36.073377 3127 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:36.073434 3127 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:36.073467 3127 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:36.082288 3127 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.130:42851
I20240502 07:59:36.082329 3255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.130:42851 every 8 connection(s)
I20240502 07:59:36.083266 3127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 07:59:36.085583 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 3127
I20240502 07:59:36.085732 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 07:59:36.088021 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.131:0
--local_ip_for_outbound_sockets=127.18.13.131
--webserver_interface=127.18.13.131
--webserver_port=0
--tserver_master_addrs=127.18.13.190:45049
--builtin_ntp_servers=127.18.13.148:37855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:36.092370 3256 heartbeater.cc:347] Connected to a master server at 127.18.13.190:45049
I20240502 07:59:36.092597 3256 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:36.093099 3256 heartbeater.cc:511] Master 127.18.13.190:45049 requested a full tablet report, sending...
I20240502 07:59:36.134016 2936 ts_manager.cc:194] Registered new tserver with Master: 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:36.134968 2936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.130:42841
W20240502 07:59:36.203370 3259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:36.203580 3259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:36.203624 3259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:36.206631 3259 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:36.206691 3259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:36.206785 3259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.131
I20240502 07:59:36.210302 3259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:37855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.18.13.131
--webserver_port=0
--tserver_master_addrs=127.18.13.190:45049
--never_fsync=true
--heap_profile_path=/tmp/kudu.3259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.131
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:36.211179 3259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:36.212433 3259 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:36.221020 3268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:36.221469 3265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:36.222281 3266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:36.223023 3259 server_base.cc:1034] running on GCE node
I20240502 07:59:36.223461 3259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:36.224226 3259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:36.225422 3259 hybrid_clock.cc:648] HybridClock initialized: now 1714636776225401 us; error 37 us; skew 500 ppm
I20240502 07:59:36.227648 3259 webserver.cc:460] Webserver started at http://127.18.13.131:42859/ using document root <none> and password file <none>
I20240502 07:59:36.228286 3259 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:36.228361 3259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:36.228591 3259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:36.230353 3259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e51a15732ba5427cbe7cf7466dbf9274"
format_stamp: "Formatted at 2024-05-02 07:59:36 on dist-test-slave-f46z"
I20240502 07:59:36.230877 3259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e51a15732ba5427cbe7cf7466dbf9274"
format_stamp: "Formatted at 2024-05-02 07:59:36 on dist-test-slave-f46z"
I20240502 07:59:36.234566 3259 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240502 07:59:36.236896 3274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:36.238026 3259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:36.238152 3259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e51a15732ba5427cbe7cf7466dbf9274"
format_stamp: "Formatted at 2024-05-02 07:59:36 on dist-test-slave-f46z"
I20240502 07:59:36.238240 3259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:36.269474 3259 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:36.270186 3259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:36.270363 3259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:36.270948 3259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:36.271890 3259 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:36.271940 3259 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:36.271999 3259 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:36.272032 3259 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:36.280437 3259 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.131:43705
I20240502 07:59:36.280485 3387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.131:43705 every 8 connection(s)
I20240502 07:59:36.281395 3259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 07:59:36.286283 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 3259
I20240502 07:59:36.286422 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 07:59:36.288798 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.132:0
--local_ip_for_outbound_sockets=127.18.13.132
--webserver_interface=127.18.13.132
--webserver_port=0
--tserver_master_addrs=127.18.13.190:45049
--builtin_ntp_servers=127.18.13.148:37855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:36.290644 3388 heartbeater.cc:347] Connected to a master server at 127.18.13.190:45049
I20240502 07:59:36.290886 3388 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:36.291445 3388 heartbeater.cc:511] Master 127.18.13.190:45049 requested a full tablet report, sending...
I20240502 07:59:36.325948 2936 ts_manager.cc:194] Registered new tserver with Master: e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:36.326685 2936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.131:50261
W20240502 07:59:36.399133 3391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:36.399392 3391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:36.399477 3391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:36.402585 3391 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:36.402667 3391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:36.402768 3391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.132
I20240502 07:59:36.406446 3391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:37855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.18.13.132
--webserver_port=0
--tserver_master_addrs=127.18.13.190:45049
--never_fsync=true
--heap_profile_path=/tmp/kudu.3391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.132
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:36.407368 3391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:36.408537 3391 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:36.416962 3397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:36.417500 3400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:36.418246 3398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:36.418987 3391 server_base.cc:1034] running on GCE node
I20240502 07:59:36.419366 3391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:36.419958 3391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:36.421119 3391 hybrid_clock.cc:648] HybridClock initialized: now 1714636776421105 us; error 27 us; skew 500 ppm
I20240502 07:59:36.423038 3391 webserver.cc:460] Webserver started at http://127.18.13.132:42721/ using document root <none> and password file <none>
I20240502 07:59:36.423571 3391 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:36.423624 3391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:36.423832 3391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:36.425554 3391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "73bb095f4df448ae81da20d1a671547c"
format_stamp: "Formatted at 2024-05-02 07:59:36 on dist-test-slave-f46z"
I20240502 07:59:36.426012 3391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "73bb095f4df448ae81da20d1a671547c"
format_stamp: "Formatted at 2024-05-02 07:59:36 on dist-test-slave-f46z"
I20240502 07:59:36.429471 3391 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:36.431710 3406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:36.432773 3391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:36.432883 3391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "73bb095f4df448ae81da20d1a671547c"
format_stamp: "Formatted at 2024-05-02 07:59:36 on dist-test-slave-f46z"
I20240502 07:59:36.432966 3391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:36.458019 3391 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:36.458653 3391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:36.458797 3391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:36.459313 3391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:36.460224 3391 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:36.460273 3391 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:36.460346 3391 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:36.460368 3391 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:36.468930 3391 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.132:35129
I20240502 07:59:36.468951 3519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.132:35129 every 8 connection(s)
I20240502 07:59:36.469897 3391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:59:36.476377 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 3391
I20240502 07:59:36.476501 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 07:59:36.478776 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.133:0
--local_ip_for_outbound_sockets=127.18.13.133
--webserver_interface=127.18.13.133
--webserver_port=0
--tserver_master_addrs=127.18.13.190:45049
--builtin_ntp_servers=127.18.13.148:37855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:36.479391 3520 heartbeater.cc:347] Connected to a master server at 127.18.13.190:45049
I20240502 07:59:36.479643 3520 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:36.480357 3520 heartbeater.cc:511] Master 127.18.13.190:45049 requested a full tablet report, sending...
I20240502 07:59:36.514137 2936 ts_manager.cc:194] Registered new tserver with Master: 73bb095f4df448ae81da20d1a671547c (127.18.13.132:35129)
I20240502 07:59:36.515053 2936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.132:41245
W20240502 07:59:36.593322 3523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:36.593552 3523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:36.593595 3523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:36.596731 3523 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:36.596817 3523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:36.596932 3523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.133
I20240502 07:59:36.600946 3523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:37855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.18.13.133
--webserver_port=0
--tserver_master_addrs=127.18.13.190:45049
--never_fsync=true
--heap_profile_path=/tmp/kudu.3523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.133
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:36.601899 3523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:36.603016 3523 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:36.611970 3530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:36.612458 3532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:36.613081 3523 server_base.cc:1034] running on GCE node
W20240502 07:59:36.613762 3529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:36.614188 3523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:36.614789 3523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:36.615934 3523 hybrid_clock.cc:648] HybridClock initialized: now 1714636776615913 us; error 40 us; skew 500 ppm
I20240502 07:59:36.617974 3523 webserver.cc:460] Webserver started at http://127.18.13.133:34207/ using document root <none> and password file <none>
I20240502 07:59:36.618594 3523 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:36.618656 3523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:36.618826 3523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:36.620559 3523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "2a45c4829a5d41c4bc1c1e9c91277264"
format_stamp: "Formatted at 2024-05-02 07:59:36 on dist-test-slave-f46z"
I20240502 07:59:36.621006 3523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "2a45c4829a5d41c4bc1c1e9c91277264"
format_stamp: "Formatted at 2024-05-02 07:59:36 on dist-test-slave-f46z"
I20240502 07:59:36.624378 3523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 07:59:36.626574 3538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:36.627640 3523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:36.627758 3523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2a45c4829a5d41c4bc1c1e9c91277264"
format_stamp: "Formatted at 2024-05-02 07:59:36 on dist-test-slave-f46z"
I20240502 07:59:36.627898 3523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:36.648105 3523 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:36.648733 3523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:36.648878 3523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:36.649385 3523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:36.650264 3523 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:36.650311 3523 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:36.650352 3523 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:36.650380 3523 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:36.658740 3523 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.133:40979
I20240502 07:59:36.658771 3651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.133:40979 every 8 connection(s)
I20240502 07:59:36.659662 3523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:59:36.666833 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 3523
I20240502 07:59:36.666968 18486 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 07:59:36.669070 3652 heartbeater.cc:347] Connected to a master server at 127.18.13.190:45049
I20240502 07:59:36.669314 3652 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:36.669924 3652 heartbeater.cc:511] Master 127.18.13.190:45049 requested a full tablet report, sending...
I20240502 07:59:36.706200 2935 ts_manager.cc:194] Registered new tserver with Master: 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979)
I20240502 07:59:36.706957 2935 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.133:53381
I20240502 07:59:36.718690 18486 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 07:59:36.728688 18486 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 07:59:36.728789 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 3523
I20240502 07:59:36.734544 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 3391
I20240502 07:59:36.955189 3124 heartbeater.cc:503] Master 127.18.13.190:45049 was elected leader, sending a full tablet report...
I20240502 07:59:37.137740 3256 heartbeater.cc:503] Master 127.18.13.190:45049 was elected leader, sending a full tablet report...
I20240502 07:59:37.329026 3388 heartbeater.cc:503] Master 127.18.13.190:45049 was elected leader, sending a full tablet report...
I20240502 07:59:40.492228 2935 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42952:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 07:59:40.495565 2935 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 07:59:40.513656 3322 tablet_service.cc:1467] Processing CreateTablet for tablet 0e10362e44ff490da447b508aedf4226 (DEFAULT_TABLE table=rebalance_test_table_0 [id=38b61401424d44728fb0d5830c0766fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:40.513657 3056 tablet_service.cc:1467] Processing CreateTablet for tablet 8229b8193e454089b8e7e72973fe5e65 (DEFAULT_TABLE table=rebalance_test_table_0 [id=38b61401424d44728fb0d5830c0766fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:40.513657 3058 tablet_service.cc:1467] Processing CreateTablet for tablet 0e10362e44ff490da447b508aedf4226 (DEFAULT_TABLE table=rebalance_test_table_0 [id=38b61401424d44728fb0d5830c0766fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:40.514772 3056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8229b8193e454089b8e7e72973fe5e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:40.514772 3322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e10362e44ff490da447b508aedf4226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:40.515345 3057 tablet_service.cc:1467] Processing CreateTablet for tablet c4ff93bab11949b0bdb739f5375e4d53 (DEFAULT_TABLE table=rebalance_test_table_0 [id=38b61401424d44728fb0d5830c0766fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:40.515647 3057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ff93bab11949b0bdb739f5375e4d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:40.513831 3320 tablet_service.cc:1467] Processing CreateTablet for tablet 8229b8193e454089b8e7e72973fe5e65 (DEFAULT_TABLE table=rebalance_test_table_0 [id=38b61401424d44728fb0d5830c0766fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:40.513656 3321 tablet_service.cc:1467] Processing CreateTablet for tablet c4ff93bab11949b0bdb739f5375e4d53 (DEFAULT_TABLE table=rebalance_test_table_0 [id=38b61401424d44728fb0d5830c0766fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:40.517328 3058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e10362e44ff490da447b508aedf4226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:40.517236 3188 tablet_service.cc:1467] Processing CreateTablet for tablet 8229b8193e454089b8e7e72973fe5e65 (DEFAULT_TABLE table=rebalance_test_table_0 [id=38b61401424d44728fb0d5830c0766fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:40.518257 3188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8229b8193e454089b8e7e72973fe5e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:40.517360 3190 tablet_service.cc:1467] Processing CreateTablet for tablet 0e10362e44ff490da447b508aedf4226 (DEFAULT_TABLE table=rebalance_test_table_0 [id=38b61401424d44728fb0d5830c0766fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:40.519399 3320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8229b8193e454089b8e7e72973fe5e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:40.519513 3190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e10362e44ff490da447b508aedf4226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:40.519819 3321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ff93bab11949b0bdb739f5375e4d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:40.517241 3189 tablet_service.cc:1467] Processing CreateTablet for tablet c4ff93bab11949b0bdb739f5375e4d53 (DEFAULT_TABLE table=rebalance_test_table_0 [id=38b61401424d44728fb0d5830c0766fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:40.520206 3189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ff93bab11949b0bdb739f5375e4d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:40.526655 3671 tablet_bootstrap.cc:492] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5: Bootstrap starting.
I20240502 07:59:40.527384 3672 tablet_bootstrap.cc:492] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c: Bootstrap starting.
I20240502 07:59:40.527895 3673 tablet_bootstrap.cc:492] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274: Bootstrap starting.
I20240502 07:59:40.529095 3671 tablet_bootstrap.cc:654] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:40.530037 3671 log.cc:824] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:40.530133 3672 tablet_bootstrap.cc:654] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:40.530264 3673 tablet_bootstrap.cc:654] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:40.530882 3672 log.cc:824] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:40.530898 3673 log.cc:824] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:40.532611 3671 tablet_bootstrap.cc:492] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5: No bootstrap required, opened a new log
I20240502 07:59:40.532829 3671 ts_tablet_manager.cc:1367] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 07:59:40.532986 3673 tablet_bootstrap.cc:492] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274: No bootstrap required, opened a new log
I20240502 07:59:40.533008 3672 tablet_bootstrap.cc:492] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c: No bootstrap required, opened a new log
I20240502 07:59:40.533165 3673 ts_tablet_manager.cc:1367] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 07:59:40.533180 3672 ts_tablet_manager.cc:1367] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 07:59:40.535621 3672 raft_consensus.cc:348] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.535899 3672 raft_consensus.cc:374] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:40.535980 3672 raft_consensus.cc:729] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc087b5db164c89bea2988820a9b31c, State: Initialized, Role: FOLLOWER
I20240502 07:59:40.536243 3673 raft_consensus.cc:348] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.536244 3671 raft_consensus.cc:348] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } }
I20240502 07:59:40.536482 3671 raft_consensus.cc:374] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:40.536475 3673 raft_consensus.cc:374] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:40.536538 3673 raft_consensus.cc:729] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Initialized, Role: FOLLOWER
I20240502 07:59:40.536463 3672 consensus_queue.cc:260] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.536581 3671 raft_consensus.cc:729] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11d320786dd4f1094ce083e43bfd5b5, State: Initialized, Role: FOLLOWER
I20240502 07:59:40.536988 3673 consensus_queue.cc:260] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.536985 3671 consensus_queue.cc:260] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } }
I20240502 07:59:40.537428 3672 ts_tablet_manager.cc:1398] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 07:59:40.537739 3672 tablet_bootstrap.cc:492] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: Bootstrap starting.
I20240502 07:59:40.537866 3671 ts_tablet_manager.cc:1398] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 07:59:40.537866 3673 ts_tablet_manager.cc:1398] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 07:59:40.538236 3673 tablet_bootstrap.cc:492] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274: Bootstrap starting.
I20240502 07:59:40.539316 3673 tablet_bootstrap.cc:654] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:40.540650 3673 tablet_bootstrap.cc:492] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274: No bootstrap required, opened a new log
I20240502 07:59:40.540773 3673 ts_tablet_manager.cc:1367] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:40.541060 3671 tablet_bootstrap.cc:492] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5: Bootstrap starting.
I20240502 07:59:40.541385 3673 raft_consensus.cc:348] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.541559 3673 raft_consensus.cc:374] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:40.541616 3673 raft_consensus.cc:729] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Initialized, Role: FOLLOWER
I20240502 07:59:40.541755 3673 consensus_queue.cc:260] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.542062 3673 ts_tablet_manager.cc:1398] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:40.542172 3671 tablet_bootstrap.cc:654] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:40.542272 3673 tablet_bootstrap.cc:492] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274: Bootstrap starting.
I20240502 07:59:40.543169 3673 tablet_bootstrap.cc:654] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:40.543965 3671 tablet_bootstrap.cc:492] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5: No bootstrap required, opened a new log
I20240502 07:59:40.544106 3671 ts_tablet_manager.cc:1367] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 07:59:40.544814 3671 raft_consensus.cc:348] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.544929 3671 raft_consensus.cc:374] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:40.544989 3671 raft_consensus.cc:729] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11d320786dd4f1094ce083e43bfd5b5, State: Initialized, Role: FOLLOWER
I20240502 07:59:40.545140 3671 consensus_queue.cc:260] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.545262 3672 tablet_bootstrap.cc:654] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:40.545491 3671 ts_tablet_manager.cc:1398] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:40.545706 3671 tablet_bootstrap.cc:492] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5: Bootstrap starting.
I20240502 07:59:40.545725 3673 tablet_bootstrap.cc:492] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274: No bootstrap required, opened a new log
I20240502 07:59:40.545814 3673 ts_tablet_manager.cc:1367] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:40.546700 3671 tablet_bootstrap.cc:654] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:40.546922 3672 tablet_bootstrap.cc:492] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: No bootstrap required, opened a new log
I20240502 07:59:40.547187 3672 ts_tablet_manager.cc:1367] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 07:59:40.547641 3673 raft_consensus.cc:348] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } }
I20240502 07:59:40.547827 3673 raft_consensus.cc:374] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:40.547881 3673 raft_consensus.cc:729] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Initialized, Role: FOLLOWER
I20240502 07:59:40.548033 3673 consensus_queue.cc:260] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } }
I20240502 07:59:40.548379 3673 ts_tablet_manager.cc:1398] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:40.548328 3672 raft_consensus.cc:348] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.548435 3672 raft_consensus.cc:374] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:40.548477 3672 raft_consensus.cc:729] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc087b5db164c89bea2988820a9b31c, State: Initialized, Role: FOLLOWER
I20240502 07:59:40.548633 3672 consensus_queue.cc:260] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.548980 3672 ts_tablet_manager.cc:1398] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:40.549170 3672 tablet_bootstrap.cc:492] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c: Bootstrap starting.
I20240502 07:59:40.550202 3672 tablet_bootstrap.cc:654] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:40.551824 3671 tablet_bootstrap.cc:492] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5: No bootstrap required, opened a new log
I20240502 07:59:40.551928 3671 ts_tablet_manager.cc:1367] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240502 07:59:40.551940 3672 tablet_bootstrap.cc:492] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c: No bootstrap required, opened a new log
I20240502 07:59:40.552026 3672 ts_tablet_manager.cc:1367] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:40.552382 3671 raft_consensus.cc:348] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.552423 3672 raft_consensus.cc:348] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } }
I20240502 07:59:40.552496 3671 raft_consensus.cc:374] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:40.552520 3671 raft_consensus.cc:729] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11d320786dd4f1094ce083e43bfd5b5, State: Initialized, Role: FOLLOWER
I20240502 07:59:40.552513 3672 raft_consensus.cc:374] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:40.552548 3672 raft_consensus.cc:729] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc087b5db164c89bea2988820a9b31c, State: Initialized, Role: FOLLOWER
I20240502 07:59:40.552639 3671 consensus_queue.cc:260] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.552687 3672 consensus_queue.cc:260] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } }
I20240502 07:59:40.552938 3671 ts_tablet_manager.cc:1398] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:40.553023 3672 ts_tablet_manager.cc:1398] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:40.582059 3677 raft_consensus.cc:482] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:40.582265 3677 raft_consensus.cc:504] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.583422 3677 leader_election.cc:288] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
W20240502 07:59:40.587891 3257 tablet.cc:2369] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:40.588191 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ff93bab11949b0bdb739f5375e4d53" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5" is_pre_election: true
I20240502 07:59:40.588372 3342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ff93bab11949b0bdb739f5375e4d53" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274" is_pre_election: true
I20240502 07:59:40.588589 3078 raft_consensus.cc:2454] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 0.
I20240502 07:59:40.588634 3342 raft_consensus.cc:2454] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 0.
I20240502 07:59:40.589133 3146 leader_election.cc:302] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, e51a15732ba5427cbe7cf7466dbf9274; no voters:
I20240502 07:59:40.589406 3677 raft_consensus.cc:2781] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:40.589483 3677 raft_consensus.cc:482] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:40.589532 3677 raft_consensus.cc:3037] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:40.590462 3677 raft_consensus.cc:504] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.590808 3677 leader_election.cc:288] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 election: Requested vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:40.591230 3342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ff93bab11949b0bdb739f5375e4d53" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
I20240502 07:59:40.591231 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ff93bab11949b0bdb739f5375e4d53" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5"
I20240502 07:59:40.591348 3078 raft_consensus.cc:3037] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:40.591389 3342 raft_consensus.cc:3037] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:40.592340 3078 raft_consensus.cc:2454] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 1.
I20240502 07:59:40.592442 3342 raft_consensus.cc:2454] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 1.
I20240502 07:59:40.592725 3144 leader_election.cc:302] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, d11d320786dd4f1094ce083e43bfd5b5; no voters:
I20240502 07:59:40.592936 3677 raft_consensus.cc:2781] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:40.593176 3677 raft_consensus.cc:686] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Becoming Leader. State: Replica: 3dc087b5db164c89bea2988820a9b31c, State: Running, Role: LEADER
I20240502 07:59:40.593519 3677 consensus_queue.cc:237] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.596822 2935 catalog_manager.cc:5484] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dc087b5db164c89bea2988820a9b31c (127.18.13.130). New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:40.636350 3678 raft_consensus.cc:482] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:40.636502 3678 raft_consensus.cc:504] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } }
I20240502 07:59:40.637586 3678 leader_election.cc:288] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:40.638264 3679 raft_consensus.cc:482] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:40.638402 3679 raft_consensus.cc:504] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } }
I20240502 07:59:40.639643 3679 leader_election.cc:288] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705), 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:40.642999 3210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e10362e44ff490da447b508aedf4226" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc087b5db164c89bea2988820a9b31c" is_pre_election: true
I20240502 07:59:40.642982 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e10362e44ff490da447b508aedf4226" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5" is_pre_election: true
I20240502 07:59:40.643152 3078 raft_consensus.cc:2454] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e51a15732ba5427cbe7cf7466dbf9274 in term 0.
I20240502 07:59:40.643200 3210 raft_consensus.cc:2454] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e51a15732ba5427cbe7cf7466dbf9274 in term 0.
I20240502 07:59:40.643532 3276 leader_election.cc:302] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d11d320786dd4f1094ce083e43bfd5b5, e51a15732ba5427cbe7cf7466dbf9274; no voters:
I20240502 07:59:40.643970 3678 raft_consensus.cc:2781] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:40.644044 3678 raft_consensus.cc:482] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:40.644088 3678 raft_consensus.cc:3037] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:40.644697 3342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e10362e44ff490da447b508aedf4226" candidate_uuid: "d11d320786dd4f1094ce083e43bfd5b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274" is_pre_election: true
I20240502 07:59:40.644934 3210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e10362e44ff490da447b508aedf4226" candidate_uuid: "d11d320786dd4f1094ce083e43bfd5b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc087b5db164c89bea2988820a9b31c" is_pre_election: true
I20240502 07:59:40.645071 3210 raft_consensus.cc:2454] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d11d320786dd4f1094ce083e43bfd5b5 in term 0.
I20240502 07:59:40.645195 3678 raft_consensus.cc:504] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } }
I20240502 07:59:40.645444 3342 raft_consensus.cc:2379] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d11d320786dd4f1094ce083e43bfd5b5 in current term 1: Already voted for candidate e51a15732ba5427cbe7cf7466dbf9274 in this term.
I20240502 07:59:40.645403 3014 leader_election.cc:302] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, d11d320786dd4f1094ce083e43bfd5b5; no voters:
I20240502 07:59:40.645608 3678 leader_election.cc:288] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 election: Requested vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:40.645684 3679 raft_consensus.cc:2781] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:40.645747 3679 raft_consensus.cc:482] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:40.645810 3679 raft_consensus.cc:3037] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:40.645987 3210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e10362e44ff490da447b508aedf4226" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc087b5db164c89bea2988820a9b31c"
I20240502 07:59:40.646076 3210 raft_consensus.cc:3037] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Advancing to term 1
W20240502 07:59:40.646591 3125 tablet.cc:2369] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:40.646670 3679 raft_consensus.cc:504] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } }
I20240502 07:59:40.647035 3679 leader_election.cc:288] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [CANDIDATE]: Term 1 election: Requested vote from peers e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705), 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:40.647038 3210 raft_consensus.cc:2454] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e51a15732ba5427cbe7cf7466dbf9274 in term 1.
I20240502 07:59:40.647361 3278 leader_election.cc:302] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, e51a15732ba5427cbe7cf7466dbf9274; no voters:
I20240502 07:59:40.647394 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e10362e44ff490da447b508aedf4226" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5"
I20240502 07:59:40.647584 3078 raft_consensus.cc:2379] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e51a15732ba5427cbe7cf7466dbf9274 in current term 1: Already voted for candidate d11d320786dd4f1094ce083e43bfd5b5 in this term.
I20240502 07:59:40.647552 3210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e10362e44ff490da447b508aedf4226" candidate_uuid: "d11d320786dd4f1094ce083e43bfd5b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc087b5db164c89bea2988820a9b31c"
I20240502 07:59:40.647599 3678 raft_consensus.cc:2781] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:40.647708 3210 raft_consensus.cc:2379] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d11d320786dd4f1094ce083e43bfd5b5 in current term 1: Already voted for candidate e51a15732ba5427cbe7cf7466dbf9274 in this term.
I20240502 07:59:40.647859 3678 raft_consensus.cc:686] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Becoming Leader. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Running, Role: LEADER
I20240502 07:59:40.647933 3342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e10362e44ff490da447b508aedf4226" candidate_uuid: "d11d320786dd4f1094ce083e43bfd5b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
I20240502 07:59:40.648202 3678 consensus_queue.cc:237] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } }
I20240502 07:59:40.648509 3014 leader_election.cc:302] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d11d320786dd4f1094ce083e43bfd5b5; no voters: 3dc087b5db164c89bea2988820a9b31c, e51a15732ba5427cbe7cf7466dbf9274
I20240502 07:59:40.648746 3679 raft_consensus.cc:2726] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 07:59:40.651546 2935 catalog_manager.cc:5484] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: term changed from 0 to 1, leader changed from <none> to e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131). New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:40.762606 3678 raft_consensus.cc:482] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:40.762761 3678 raft_consensus.cc:504] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.763284 3678 leader_election.cc:288] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:40.763757 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229b8193e454089b8e7e72973fe5e65" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5" is_pre_election: true
I20240502 07:59:40.763893 3210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229b8193e454089b8e7e72973fe5e65" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc087b5db164c89bea2988820a9b31c" is_pre_election: true
I20240502 07:59:40.763960 3078 raft_consensus.cc:2454] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e51a15732ba5427cbe7cf7466dbf9274 in term 0.
I20240502 07:59:40.764043 3210 raft_consensus.cc:2454] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e51a15732ba5427cbe7cf7466dbf9274 in term 0.
I20240502 07:59:40.764353 3276 leader_election.cc:302] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d11d320786dd4f1094ce083e43bfd5b5, e51a15732ba5427cbe7cf7466dbf9274; no voters:
I20240502 07:59:40.764703 3678 raft_consensus.cc:2781] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:40.764772 3678 raft_consensus.cc:482] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:40.764817 3678 raft_consensus.cc:3037] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:40.765702 3678 raft_consensus.cc:504] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.766091 3678 leader_election.cc:288] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 election: Requested vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:40.766590 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229b8193e454089b8e7e72973fe5e65" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5"
I20240502 07:59:40.766590 3210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229b8193e454089b8e7e72973fe5e65" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc087b5db164c89bea2988820a9b31c"
I20240502 07:59:40.766739 3078 raft_consensus.cc:3037] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:40.766739 3210 raft_consensus.cc:3037] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:40.767752 3210 raft_consensus.cc:2454] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e51a15732ba5427cbe7cf7466dbf9274 in term 1.
I20240502 07:59:40.767752 3078 raft_consensus.cc:2454] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e51a15732ba5427cbe7cf7466dbf9274 in term 1.
I20240502 07:59:40.768142 3278 leader_election.cc:302] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, e51a15732ba5427cbe7cf7466dbf9274; no voters:
I20240502 07:59:40.768321 3678 raft_consensus.cc:2781] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:40.768417 3678 raft_consensus.cc:686] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Becoming Leader. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Running, Role: LEADER
I20240502 07:59:40.768581 3678 consensus_queue.cc:237] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:40.770390 2935 catalog_manager.cc:5484] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: term changed from 0 to 1, leader changed from <none> to e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131). New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: HEALTHY } } }
W20240502 07:59:40.785980 3389 tablet.cc:2369] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 07:59:40.915573 3698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:40.915941 3698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:40.919229 3698 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 07:59:41.005829 3677 consensus_queue.cc:1035] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:41.012848 3688 consensus_queue.cc:1035] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:41.023404 3210 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:41.023862 3078 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:41.024128 3695 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:41.024487 3678 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:41.032599 3077 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:41.033202 3678 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:41.035620 3209 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:41.038719 3678 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:41.043099 3727 mvcc.cc:205] Tried to move back new op lower bound from 7023152255101960192 to 7023152253537587200. Current Snapshot: MvccSnapshot[applied={T|T < 7023152255101960192}]
I20240502 07:59:41.049181 3725 mvcc.cc:205] Tried to move back new op lower bound from 7023152255101960192 to 7023152253537587200. Current Snapshot: MvccSnapshot[applied={T|T < 7023152255101960192}]
I20240502 07:59:41.055091 3728 mvcc.cc:205] Tried to move back new op lower bound from 7023152255063355392 to 7023152254029598720. Current Snapshot: MvccSnapshot[applied={T|T < 7023152255063355392}]
INSERT report
rows total: 2000
time total: 129.215 ms
time per row: 0.0646073 ms
I20240502 07:59:41.103045 2929 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42952:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 07:59:41.103649 2929 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 07:59:41.109431 3188 tablet_service.cc:1467] Processing CreateTablet for tablet 3429dee9c9e24927992401f2af945375 (DEFAULT_TABLE table=rebalance_test_table_1 [id=90a1f5d24e6f413eb4ef048ae0e4d46e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.109580 3056 tablet_service.cc:1467] Processing CreateTablet for tablet 3429dee9c9e24927992401f2af945375 (DEFAULT_TABLE table=rebalance_test_table_1 [id=90a1f5d24e6f413eb4ef048ae0e4d46e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.109727 3057 tablet_service.cc:1467] Processing CreateTablet for tablet ee8e92d37ee14bfcbf0dd48ce054a339 (DEFAULT_TABLE table=rebalance_test_table_1 [id=90a1f5d24e6f413eb4ef048ae0e4d46e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.109815 3188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3429dee9c9e24927992401f2af945375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.109874 3056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3429dee9c9e24927992401f2af945375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.110019 3057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8e92d37ee14bfcbf0dd48ce054a339. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.111654 3058 tablet_service.cc:1467] Processing CreateTablet for tablet cdfb5d62286445b98e04d397fa8ffdb1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=90a1f5d24e6f413eb4ef048ae0e4d46e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.111995 3058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfb5d62286445b98e04d397fa8ffdb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.112367 3752 tablet_bootstrap.cc:492] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5: Bootstrap starting.
I20240502 07:59:41.112525 3751 tablet_bootstrap.cc:492] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c: Bootstrap starting.
I20240502 07:59:41.113047 3190 tablet_service.cc:1467] Processing CreateTablet for tablet cdfb5d62286445b98e04d397fa8ffdb1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=90a1f5d24e6f413eb4ef048ae0e4d46e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.113307 3190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfb5d62286445b98e04d397fa8ffdb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.113338 3752 tablet_bootstrap.cc:654] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.113551 3751 tablet_bootstrap.cc:654] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.114292 3188 tablet_service.cc:1467] Processing CreateTablet for tablet ee8e92d37ee14bfcbf0dd48ce054a339 (DEFAULT_TABLE table=rebalance_test_table_1 [id=90a1f5d24e6f413eb4ef048ae0e4d46e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.114534 3188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8e92d37ee14bfcbf0dd48ce054a339. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.114560 3322 tablet_service.cc:1467] Processing CreateTablet for tablet 3429dee9c9e24927992401f2af945375 (DEFAULT_TABLE table=rebalance_test_table_1 [id=90a1f5d24e6f413eb4ef048ae0e4d46e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.114845 3322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3429dee9c9e24927992401f2af945375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.115396 3752 tablet_bootstrap.cc:492] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5: No bootstrap required, opened a new log
I20240502 07:59:41.115542 3752 ts_tablet_manager.cc:1367] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:41.116151 3752 raft_consensus.cc:348] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.116293 3752 raft_consensus.cc:374] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.116329 3752 raft_consensus.cc:729] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11d320786dd4f1094ce083e43bfd5b5, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.116467 3752 consensus_queue.cc:260] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.116787 3752 ts_tablet_manager.cc:1398] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:41.116950 3752 tablet_bootstrap.cc:492] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5: Bootstrap starting.
I20240502 07:59:41.117255 3320 tablet_service.cc:1467] Processing CreateTablet for tablet ee8e92d37ee14bfcbf0dd48ce054a339 (DEFAULT_TABLE table=rebalance_test_table_1 [id=90a1f5d24e6f413eb4ef048ae0e4d46e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.117509 3320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8e92d37ee14bfcbf0dd48ce054a339. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.117559 3321 tablet_service.cc:1467] Processing CreateTablet for tablet cdfb5d62286445b98e04d397fa8ffdb1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=90a1f5d24e6f413eb4ef048ae0e4d46e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.117807 3321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfb5d62286445b98e04d397fa8ffdb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.117957 3752 tablet_bootstrap.cc:654] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.120031 3755 tablet_bootstrap.cc:492] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274: Bootstrap starting.
I20240502 07:59:41.120855 3751 tablet_bootstrap.cc:492] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c: No bootstrap required, opened a new log
I20240502 07:59:41.120983 3752 tablet_bootstrap.cc:492] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5: No bootstrap required, opened a new log
I20240502 07:59:41.120997 3751 ts_tablet_manager.cc:1367] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 07:59:41.121095 3752 ts_tablet_manager.cc:1367] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:41.121110 3755 tablet_bootstrap.cc:654] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.121537 3752 raft_consensus.cc:348] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.121688 3752 raft_consensus.cc:374] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.121730 3752 raft_consensus.cc:729] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11d320786dd4f1094ce083e43bfd5b5, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.121650 3751 raft_consensus.cc:348] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.121861 3751 raft_consensus.cc:374] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.121913 3751 raft_consensus.cc:729] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc087b5db164c89bea2988820a9b31c, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.121879 3752 consensus_queue.cc:260] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.122076 3751 consensus_queue.cc:260] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.122375 3751 ts_tablet_manager.cc:1398] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:41.122575 3751 tablet_bootstrap.cc:492] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c: Bootstrap starting.
I20240502 07:59:41.123023 3752 ts_tablet_manager.cc:1398] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:41.123226 3752 tablet_bootstrap.cc:492] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5: Bootstrap starting.
I20240502 07:59:41.123589 3751 tablet_bootstrap.cc:654] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.124758 3752 tablet_bootstrap.cc:654] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.124790 3751 tablet_bootstrap.cc:492] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c: No bootstrap required, opened a new log
I20240502 07:59:41.124871 3751 ts_tablet_manager.cc:1367] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:41.125192 3755 tablet_bootstrap.cc:492] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274: No bootstrap required, opened a new log
I20240502 07:59:41.125327 3755 ts_tablet_manager.cc:1367] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 07:59:41.125778 3751 raft_consensus.cc:348] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.125916 3751 raft_consensus.cc:374] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.125970 3751 raft_consensus.cc:729] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc087b5db164c89bea2988820a9b31c, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.126111 3751 consensus_queue.cc:260] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.126096 3755 raft_consensus.cc:348] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.126336 3755 raft_consensus.cc:374] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.126394 3755 raft_consensus.cc:729] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.126559 3751 ts_tablet_manager.cc:1398] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:41.126543 3755 consensus_queue.cc:260] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.126816 3755 ts_tablet_manager.cc:1398] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:41.126874 3751 tablet_bootstrap.cc:492] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c: Bootstrap starting.
I20240502 07:59:41.126978 3755 tablet_bootstrap.cc:492] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274: Bootstrap starting.
I20240502 07:59:41.127949 3755 tablet_bootstrap.cc:654] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.128078 3751 tablet_bootstrap.cc:654] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.129477 3755 tablet_bootstrap.cc:492] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274: No bootstrap required, opened a new log
I20240502 07:59:41.129626 3755 ts_tablet_manager.cc:1367] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:59:41.130064 3751 tablet_bootstrap.cc:492] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c: No bootstrap required, opened a new log
I20240502 07:59:41.130175 3751 ts_tablet_manager.cc:1367] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:59:41.130129 3755 raft_consensus.cc:348] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.130255 3755 raft_consensus.cc:374] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.130298 3755 raft_consensus.cc:729] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.129606 3752 tablet_bootstrap.cc:492] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5: No bootstrap required, opened a new log
I20240502 07:59:41.130417 3752 ts_tablet_manager.cc:1367] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 07:59:41.130602 3755 consensus_queue.cc:260] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.130910 3755 ts_tablet_manager.cc:1398] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:41.130985 3751 raft_consensus.cc:348] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.130987 3752 raft_consensus.cc:348] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.131093 3752 raft_consensus.cc:374] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.131093 3751 raft_consensus.cc:374] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.131104 3755 tablet_bootstrap.cc:492] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274: Bootstrap starting.
I20240502 07:59:41.131126 3752 raft_consensus.cc:729] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11d320786dd4f1094ce083e43bfd5b5, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.131126 3751 raft_consensus.cc:729] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc087b5db164c89bea2988820a9b31c, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.131295 3752 consensus_queue.cc:260] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.131441 3751 consensus_queue.cc:260] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.131734 3751 ts_tablet_manager.cc:1398] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:41.132097 3755 tablet_bootstrap.cc:654] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.132849 3752 ts_tablet_manager.cc:1398] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:41.133394 3755 tablet_bootstrap.cc:492] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274: No bootstrap required, opened a new log
I20240502 07:59:41.133487 3755 ts_tablet_manager.cc:1367] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 07:59:41.133970 3755 raft_consensus.cc:348] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.134099 3755 raft_consensus.cc:374] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.134133 3755 raft_consensus.cc:729] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.134251 3755 consensus_queue.cc:260] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.134636 3755 ts_tablet_manager.cc:1398] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 07:59:41.134624 3679 raft_consensus.cc:482] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:41.134709 3679 raft_consensus.cc:504] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.135082 3679 leader_election.cc:288] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851), e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:41.135432 3210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfb5d62286445b98e04d397fa8ffdb1" candidate_uuid: "d11d320786dd4f1094ce083e43bfd5b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc087b5db164c89bea2988820a9b31c" is_pre_election: true
I20240502 07:59:41.135581 3210 raft_consensus.cc:2454] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d11d320786dd4f1094ce083e43bfd5b5 in term 0.
I20240502 07:59:41.135579 3342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfb5d62286445b98e04d397fa8ffdb1" candidate_uuid: "d11d320786dd4f1094ce083e43bfd5b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274" is_pre_election: true
I20240502 07:59:41.135705 3342 raft_consensus.cc:2454] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d11d320786dd4f1094ce083e43bfd5b5 in term 0.
I20240502 07:59:41.135921 3014 leader_election.cc:302] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, d11d320786dd4f1094ce083e43bfd5b5; no voters:
I20240502 07:59:41.136116 3679 raft_consensus.cc:2781] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:41.136194 3679 raft_consensus.cc:482] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:41.136234 3679 raft_consensus.cc:3037] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.137091 3679 raft_consensus.cc:504] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.137461 3679 leader_election.cc:288] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [CANDIDATE]: Term 1 election: Requested vote from peers 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851), e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:41.137907 3342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfb5d62286445b98e04d397fa8ffdb1" candidate_uuid: "d11d320786dd4f1094ce083e43bfd5b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
I20240502 07:59:41.137874 3210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfb5d62286445b98e04d397fa8ffdb1" candidate_uuid: "d11d320786dd4f1094ce083e43bfd5b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc087b5db164c89bea2988820a9b31c"
I20240502 07:59:41.138015 3342 raft_consensus.cc:3037] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.138025 3210 raft_consensus.cc:3037] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.138926 3210 raft_consensus.cc:2454] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d11d320786dd4f1094ce083e43bfd5b5 in term 1.
I20240502 07:59:41.139176 3342 raft_consensus.cc:2454] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d11d320786dd4f1094ce083e43bfd5b5 in term 1.
I20240502 07:59:41.139246 3014 leader_election.cc:302] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, d11d320786dd4f1094ce083e43bfd5b5; no voters:
I20240502 07:59:41.139464 3679 raft_consensus.cc:2781] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:41.139740 3679 raft_consensus.cc:686] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 LEADER]: Becoming Leader. State: Replica: d11d320786dd4f1094ce083e43bfd5b5, State: Running, Role: LEADER
I20240502 07:59:41.140214 3679 consensus_queue.cc:237] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.143182 2929 catalog_manager.cc:5484] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129). New cstate: current_term: 1 leader_uuid: "d11d320786dd4f1094ce083e43bfd5b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:41.234020 3726 raft_consensus.cc:482] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:41.234176 3726 raft_consensus.cc:504] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.234714 3726 leader_election.cc:288] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:41.235229 3342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8e92d37ee14bfcbf0dd48ce054a339" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274" is_pre_election: true
I20240502 07:59:41.235229 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8e92d37ee14bfcbf0dd48ce054a339" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5" is_pre_election: true
I20240502 07:59:41.235395 3342 raft_consensus.cc:2454] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 0.
I20240502 07:59:41.235395 3078 raft_consensus.cc:2454] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 0.
I20240502 07:59:41.235739 3146 leader_election.cc:302] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, e51a15732ba5427cbe7cf7466dbf9274; no voters:
I20240502 07:59:41.235949 3726 raft_consensus.cc:2781] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:41.236007 3726 raft_consensus.cc:482] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:41.236035 3726 raft_consensus.cc:3037] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.237000 3726 raft_consensus.cc:504] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.237429 3726 leader_election.cc:288] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 election: Requested vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:41.237888 3342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8e92d37ee14bfcbf0dd48ce054a339" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
I20240502 07:59:41.237888 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee8e92d37ee14bfcbf0dd48ce054a339" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5"
I20240502 07:59:41.238025 3342 raft_consensus.cc:3037] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.238025 3078 raft_consensus.cc:3037] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.239050 3342 raft_consensus.cc:2454] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 1.
I20240502 07:59:41.239050 3078 raft_consensus.cc:2454] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 1.
I20240502 07:59:41.239423 3146 leader_election.cc:302] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, e51a15732ba5427cbe7cf7466dbf9274; no voters:
I20240502 07:59:41.239634 3726 raft_consensus.cc:2781] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:41.239735 3726 raft_consensus.cc:686] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Becoming Leader. State: Replica: 3dc087b5db164c89bea2988820a9b31c, State: Running, Role: LEADER
I20240502 07:59:41.239915 3726 consensus_queue.cc:237] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.241638 2929 catalog_manager.cc:5484] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dc087b5db164c89bea2988820a9b31c (127.18.13.130). New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:41.380674 3726 raft_consensus.cc:482] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:41.380826 3726 raft_consensus.cc:504] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.381373 3726 leader_election.cc:288] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:41.381845 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3429dee9c9e24927992401f2af945375" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5" is_pre_election: true
I20240502 07:59:41.381979 3078 raft_consensus.cc:2454] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 0.
I20240502 07:59:41.381913 3342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3429dee9c9e24927992401f2af945375" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274" is_pre_election: true
I20240502 07:59:41.382089 3342 raft_consensus.cc:2454] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 0.
I20240502 07:59:41.382354 3144 leader_election.cc:302] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, d11d320786dd4f1094ce083e43bfd5b5; no voters:
I20240502 07:59:41.382608 3726 raft_consensus.cc:2781] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:41.382658 3726 raft_consensus.cc:482] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:41.382684 3726 raft_consensus.cc:3037] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.383486 3726 raft_consensus.cc:504] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.383893 3726 leader_election.cc:288] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 election: Requested vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:41.384281 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3429dee9c9e24927992401f2af945375" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5"
I20240502 07:59:41.384377 3342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3429dee9c9e24927992401f2af945375" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
I20240502 07:59:41.384429 3078 raft_consensus.cc:3037] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.384518 3342 raft_consensus.cc:3037] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.385255 3078 raft_consensus.cc:2454] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 1.
I20240502 07:59:41.385293 3342 raft_consensus.cc:2454] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 1.
I20240502 07:59:41.385596 3144 leader_election.cc:302] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, d11d320786dd4f1094ce083e43bfd5b5; no voters:
I20240502 07:59:41.385763 3726 raft_consensus.cc:2781] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:41.385891 3726 raft_consensus.cc:686] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Becoming Leader. State: Replica: 3dc087b5db164c89bea2988820a9b31c, State: Running, Role: LEADER
I20240502 07:59:41.386054 3726 consensus_queue.cc:237] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.388039 2929 catalog_manager.cc:5484] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dc087b5db164c89bea2988820a9b31c (127.18.13.130). New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:41.534602 3767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:41.534917 3767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:41.538394 3767 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:59:41.545341 3679 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:41.549772 3763 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 07:59:41.624176 3726 consensus_queue.cc:1035] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:41.628171 3688 consensus_queue.cc:1035] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:41.639288 3341 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:41.640666 3078 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:41.641242 3688 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:41.645807 3726 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.022 ms
time per row: 0.0620108 ms
I20240502 07:59:41.712453 2928 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42952:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 07:59:41.713088 2928 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 07:59:41.718680 3188 tablet_service.cc:1467] Processing CreateTablet for tablet e6b4e558aad64651aad857a4cbfac9d9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=242dd44a47a143d6ad62e53b54120b3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.718883 3320 tablet_service.cc:1467] Processing CreateTablet for tablet e6b4e558aad64651aad857a4cbfac9d9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=242dd44a47a143d6ad62e53b54120b3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.719056 3188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b4e558aad64651aad857a4cbfac9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.719213 3320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b4e558aad64651aad857a4cbfac9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.719341 3058 tablet_service.cc:1467] Processing CreateTablet for tablet e6b4e558aad64651aad857a4cbfac9d9 (DEFAULT_TABLE table=rebalance_test_table_2 [id=242dd44a47a143d6ad62e53b54120b3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.719625 3058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b4e558aad64651aad857a4cbfac9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.722101 3188 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0ea65032f64a6fb74b88fb4fc92f0b (DEFAULT_TABLE table=rebalance_test_table_2 [id=242dd44a47a143d6ad62e53b54120b3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.722519 3321 tablet_service.cc:1467] Processing CreateTablet for tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 (DEFAULT_TABLE table=rebalance_test_table_2 [id=242dd44a47a143d6ad62e53b54120b3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.722656 3820 tablet_bootstrap.cc:492] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c: Bootstrap starting.
I20240502 07:59:41.722697 3058 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0ea65032f64a6fb74b88fb4fc92f0b (DEFAULT_TABLE table=rebalance_test_table_2 [id=242dd44a47a143d6ad62e53b54120b3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.722783 3321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d73ac6edc2549e7a4c7dbc2bb91d750. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.722726 3057 tablet_service.cc:1467] Processing CreateTablet for tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 (DEFAULT_TABLE table=rebalance_test_table_2 [id=242dd44a47a143d6ad62e53b54120b3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.722134 3190 tablet_service.cc:1467] Processing CreateTablet for tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 (DEFAULT_TABLE table=rebalance_test_table_2 [id=242dd44a47a143d6ad62e53b54120b3a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.722913 3058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0ea65032f64a6fb74b88fb4fc92f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.723156 3057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d73ac6edc2549e7a4c7dbc2bb91d750. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.723129 3821 tablet_bootstrap.cc:492] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274: Bootstrap starting.
I20240502 07:59:41.725140 3320 tablet_service.cc:1467] Processing CreateTablet for tablet 7f0ea65032f64a6fb74b88fb4fc92f0b (DEFAULT_TABLE table=rebalance_test_table_2 [id=242dd44a47a143d6ad62e53b54120b3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:41.725240 3822 tablet_bootstrap.cc:492] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5: Bootstrap starting.
I20240502 07:59:41.725402 3320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0ea65032f64a6fb74b88fb4fc92f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.725554 3188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0ea65032f64a6fb74b88fb4fc92f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.725764 3190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d73ac6edc2549e7a4c7dbc2bb91d750. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.726184 3822 tablet_bootstrap.cc:654] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.727381 3822 tablet_bootstrap.cc:492] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5: No bootstrap required, opened a new log
I20240502 07:59:41.727491 3822 ts_tablet_manager.cc:1367] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:41.728034 3822 raft_consensus.cc:348] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.728175 3822 raft_consensus.cc:374] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.728214 3822 raft_consensus.cc:729] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11d320786dd4f1094ce083e43bfd5b5, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.728363 3822 consensus_queue.cc:260] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.728660 3822 ts_tablet_manager.cc:1398] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:41.728843 3822 tablet_bootstrap.cc:492] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5: Bootstrap starting.
I20240502 07:59:41.729346 3820 tablet_bootstrap.cc:654] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.730007 3822 tablet_bootstrap.cc:654] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.730405 3821 tablet_bootstrap.cc:654] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.731107 3820 tablet_bootstrap.cc:492] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c: No bootstrap required, opened a new log
I20240502 07:59:41.731222 3820 ts_tablet_manager.cc:1367] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 07:59:41.731559 3821 tablet_bootstrap.cc:492] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274: No bootstrap required, opened a new log
I20240502 07:59:41.731655 3821 ts_tablet_manager.cc:1367] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240502 07:59:41.731745 3820 raft_consensus.cc:348] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.731920 3820 raft_consensus.cc:374] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.731961 3820 raft_consensus.cc:729] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc087b5db164c89bea2988820a9b31c, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.731380 3822 tablet_bootstrap.cc:492] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5: No bootstrap required, opened a new log
I20240502 07:59:41.732105 3820 consensus_queue.cc:260] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.732167 3822 ts_tablet_manager.cc:1367] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:41.732178 3821 raft_consensus.cc:348] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.732290 3821 raft_consensus.cc:374] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.732334 3821 raft_consensus.cc:729] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.732419 3820 ts_tablet_manager.cc:1398] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:41.732479 3821 consensus_queue.cc:260] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.732620 3820 tablet_bootstrap.cc:492] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c: Bootstrap starting.
I20240502 07:59:41.732770 3821 ts_tablet_manager.cc:1398] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:41.732954 3821 tablet_bootstrap.cc:492] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274: Bootstrap starting.
I20240502 07:59:41.733104 3822 raft_consensus.cc:348] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.733265 3822 raft_consensus.cc:374] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.733340 3822 raft_consensus.cc:729] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11d320786dd4f1094ce083e43bfd5b5, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.733592 3820 tablet_bootstrap.cc:654] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.733556 3822 consensus_queue.cc:260] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.733912 3821 tablet_bootstrap.cc:654] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.734030 3822 ts_tablet_manager.cc:1398] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:41.734274 3822 tablet_bootstrap.cc:492] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5: Bootstrap starting.
I20240502 07:59:41.735325 3822 tablet_bootstrap.cc:654] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.735682 3821 tablet_bootstrap.cc:492] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274: No bootstrap required, opened a new log
I20240502 07:59:41.735817 3821 ts_tablet_manager.cc:1367] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 07:59:41.735980 3820 tablet_bootstrap.cc:492] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c: No bootstrap required, opened a new log
I20240502 07:59:41.736107 3820 ts_tablet_manager.cc:1367] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:41.736333 3821 raft_consensus.cc:348] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.736438 3821 raft_consensus.cc:374] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.736470 3821 raft_consensus.cc:729] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.736605 3821 consensus_queue.cc:260] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.736603 3820 raft_consensus.cc:348] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.736706 3820 raft_consensus.cc:374] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.736744 3820 raft_consensus.cc:729] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc087b5db164c89bea2988820a9b31c, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.736900 3821 ts_tablet_manager.cc:1398] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:41.736881 3820 consensus_queue.cc:260] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.737077 3821 tablet_bootstrap.cc:492] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274: Bootstrap starting.
I20240502 07:59:41.737169 3820 ts_tablet_manager.cc:1398] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:41.737344 3820 tablet_bootstrap.cc:492] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c: Bootstrap starting.
I20240502 07:59:41.737963 3821 tablet_bootstrap.cc:654] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.738320 3820 tablet_bootstrap.cc:654] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:41.739173 3821 tablet_bootstrap.cc:492] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274: No bootstrap required, opened a new log
I20240502 07:59:41.739264 3821 ts_tablet_manager.cc:1367] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:41.739631 3820 tablet_bootstrap.cc:492] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c: No bootstrap required, opened a new log
I20240502 07:59:41.739727 3820 ts_tablet_manager.cc:1367] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 07:59:41.739676 3821 raft_consensus.cc:348] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.739825 3821 raft_consensus.cc:374] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.739868 3821 raft_consensus.cc:729] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.740031 3821 consensus_queue.cc:260] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.740293 3821 ts_tablet_manager.cc:1398] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:41.740237 3820 raft_consensus.cc:348] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.740340 3820 raft_consensus.cc:374] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.740389 3820 raft_consensus.cc:729] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc087b5db164c89bea2988820a9b31c, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.740535 3820 consensus_queue.cc:260] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.740895 3820 ts_tablet_manager.cc:1398] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:41.741144 3822 tablet_bootstrap.cc:492] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5: No bootstrap required, opened a new log
I20240502 07:59:41.741250 3822 ts_tablet_manager.cc:1367] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 07:59:41.741779 3822 raft_consensus.cc:348] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.741883 3822 raft_consensus.cc:374] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:41.741904 3822 raft_consensus.cc:729] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11d320786dd4f1094ce083e43bfd5b5, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.742012 3822 consensus_queue.cc:260] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.742295 3822 ts_tablet_manager.cc:1398] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:41.750499 3750 raft_consensus.cc:482] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:41.750677 3750 raft_consensus.cc:504] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.751219 3750 leader_election.cc:288] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:41.751715 3210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0ea65032f64a6fb74b88fb4fc92f0b" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc087b5db164c89bea2988820a9b31c" is_pre_election: true
I20240502 07:59:41.751700 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0ea65032f64a6fb74b88fb4fc92f0b" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5" is_pre_election: true
I20240502 07:59:41.751906 3078 raft_consensus.cc:2454] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e51a15732ba5427cbe7cf7466dbf9274 in term 0.
I20240502 07:59:41.752033 3210 raft_consensus.cc:2454] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e51a15732ba5427cbe7cf7466dbf9274 in term 0.
I20240502 07:59:41.752389 3278 leader_election.cc:302] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, e51a15732ba5427cbe7cf7466dbf9274; no voters:
I20240502 07:59:41.752678 3750 raft_consensus.cc:2781] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:41.752738 3750 raft_consensus.cc:482] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:41.752785 3750 raft_consensus.cc:3037] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.754153 3750 raft_consensus.cc:504] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.754626 3750 leader_election.cc:288] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 election: Requested vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:41.755165 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0ea65032f64a6fb74b88fb4fc92f0b" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5"
I20240502 07:59:41.755312 3078 raft_consensus.cc:3037] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.755355 3210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0ea65032f64a6fb74b88fb4fc92f0b" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc087b5db164c89bea2988820a9b31c"
I20240502 07:59:41.755488 3210 raft_consensus.cc:3037] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.756513 3078 raft_consensus.cc:2454] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e51a15732ba5427cbe7cf7466dbf9274 in term 1.
I20240502 07:59:41.756745 3210 raft_consensus.cc:2454] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e51a15732ba5427cbe7cf7466dbf9274 in term 1.
I20240502 07:59:41.756960 3276 leader_election.cc:302] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d11d320786dd4f1094ce083e43bfd5b5, e51a15732ba5427cbe7cf7466dbf9274; no voters:
I20240502 07:59:41.757211 3750 raft_consensus.cc:2781] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:41.757439 3750 raft_consensus.cc:686] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Becoming Leader. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Running, Role: LEADER
I20240502 07:59:41.757624 3750 consensus_queue.cc:237] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.761406 2936 catalog_manager.cc:5484] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: term changed from 0 to 1, leader changed from <none> to e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131). New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: HEALTHY } } }
I20240502 07:59:41.782670 3811 raft_consensus.cc:482] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:41.782815 3811 raft_consensus.cc:504] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.783293 3811 leader_election.cc:288] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:41.783850 3341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b4e558aad64651aad857a4cbfac9d9" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274" is_pre_election: true
I20240502 07:59:41.783987 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b4e558aad64651aad857a4cbfac9d9" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5" is_pre_election: true
I20240502 07:59:41.784034 3341 raft_consensus.cc:2454] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 0.
I20240502 07:59:41.784114 3078 raft_consensus.cc:2454] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 0.
I20240502 07:59:41.784407 3146 leader_election.cc:302] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, e51a15732ba5427cbe7cf7466dbf9274; no voters:
I20240502 07:59:41.784641 3811 raft_consensus.cc:2781] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:41.784729 3811 raft_consensus.cc:482] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:41.784795 3811 raft_consensus.cc:3037] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.786199 3811 raft_consensus.cc:504] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.786751 3811 leader_election.cc:288] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 election: Requested vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:41.787200 3341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b4e558aad64651aad857a4cbfac9d9" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
I20240502 07:59:41.787362 3341 raft_consensus.cc:3037] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.788807 3341 raft_consensus.cc:2454] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 1.
I20240502 07:59:41.789208 3146 leader_election.cc:302] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, e51a15732ba5427cbe7cf7466dbf9274; no voters:
I20240502 07:59:41.789189 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6b4e558aad64651aad857a4cbfac9d9" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5"
I20240502 07:59:41.789350 3078 raft_consensus.cc:3037] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.790570 3078 raft_consensus.cc:2454] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 1.
I20240502 07:59:41.790920 3811 raft_consensus.cc:2781] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:41.791373 3811 raft_consensus.cc:686] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Becoming Leader. State: Replica: 3dc087b5db164c89bea2988820a9b31c, State: Running, Role: LEADER
I20240502 07:59:41.791599 3811 consensus_queue.cc:237] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.793799 2936 catalog_manager.cc:5484] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dc087b5db164c89bea2988820a9b31c (127.18.13.130). New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:41.934386 3750 raft_consensus.cc:482] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:41.934551 3750 raft_consensus.cc:504] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.935011 3750 leader_election.cc:288] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:41.935591 3210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d73ac6edc2549e7a4c7dbc2bb91d750" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc087b5db164c89bea2988820a9b31c" is_pre_election: true
I20240502 07:59:41.935603 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d73ac6edc2549e7a4c7dbc2bb91d750" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5" is_pre_election: true
I20240502 07:59:41.935741 3210 raft_consensus.cc:2454] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e51a15732ba5427cbe7cf7466dbf9274 in term 0.
I20240502 07:59:41.935761 3078 raft_consensus.cc:2454] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e51a15732ba5427cbe7cf7466dbf9274 in term 0.
I20240502 07:59:41.936121 3278 leader_election.cc:302] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, e51a15732ba5427cbe7cf7466dbf9274; no voters:
I20240502 07:59:41.936304 3750 raft_consensus.cc:2781] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:41.936373 3750 raft_consensus.cc:482] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:41.936401 3750 raft_consensus.cc:3037] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.937222 3750 raft_consensus.cc:504] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.937590 3750 leader_election.cc:288] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 election: Requested vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:41.938094 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d73ac6edc2549e7a4c7dbc2bb91d750" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5"
I20240502 07:59:41.938119 3210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d73ac6edc2549e7a4c7dbc2bb91d750" candidate_uuid: "e51a15732ba5427cbe7cf7466dbf9274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc087b5db164c89bea2988820a9b31c"
I20240502 07:59:41.938243 3210 raft_consensus.cc:3037] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.938249 3078 raft_consensus.cc:3037] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:41.939113 3210 raft_consensus.cc:2454] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e51a15732ba5427cbe7cf7466dbf9274 in term 1.
I20240502 07:59:41.939134 3078 raft_consensus.cc:2454] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e51a15732ba5427cbe7cf7466dbf9274 in term 1.
I20240502 07:59:41.939440 3278 leader_election.cc:302] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, e51a15732ba5427cbe7cf7466dbf9274; no voters:
I20240502 07:59:41.939738 3750 raft_consensus.cc:2781] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:41.939934 3750 raft_consensus.cc:686] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Becoming Leader. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Running, Role: LEADER
I20240502 07:59:41.940160 3750 consensus_queue.cc:237] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } }
I20240502 07:59:41.941943 2936 catalog_manager.cc:5484] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: term changed from 0 to 1, leader changed from <none> to e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131). New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: HEALTHY } } }
W20240502 07:59:42.082471 3836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:42.082765 3836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:42.087312 3836 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 07:59:42.193006 3210 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:42.193150 3078 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:42.193467 3750 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:42.193696 3832 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:42.196003 3209 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:42.197427 3750 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:42.199743 3341 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:42.200238 3811 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:42.202636 3078 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:42.203873 3832 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:42.206396 3078 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:42.206976 3726 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 122.388 ms
time per row: 0.0611938 ms
I20240502 07:59:42.264791 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.133:40979
--local_ip_for_outbound_sockets=127.18.13.133
--tserver_master_addrs=127.18.13.190:45049
--webserver_port=34207
--webserver_interface=127.18.13.133
--builtin_ntp_servers=127.18.13.148:37855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 07:59:42.381165 3892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:42.381433 3892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:42.381479 3892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:42.384790 3892 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:42.384887 3892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:42.384960 3892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.133
I20240502 07:59:42.388928 3892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:37855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.133:40979
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.18.13.133
--webserver_port=34207
--tserver_master_addrs=127.18.13.190:45049
--never_fsync=true
--heap_profile_path=/tmp/kudu.3892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.133
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:42.389932 3892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:42.391302 3892 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:42.400919 3900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:42.401614 3897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:42.402370 3892 server_base.cc:1034] running on GCE node
W20240502 07:59:42.402774 3898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:42.403271 3892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:42.403959 3892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:42.405128 3892 hybrid_clock.cc:648] HybridClock initialized: now 1714636782405100 us; error 43 us; skew 500 ppm
I20240502 07:59:42.407277 3892 webserver.cc:460] Webserver started at http://127.18.13.133:34207/ using document root <none> and password file <none>
I20240502 07:59:42.407912 3892 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:42.407995 3892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:42.411628 3892 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:42.413789 3906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:42.414953 3892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:42.415084 3892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2a45c4829a5d41c4bc1c1e9c91277264"
format_stamp: "Formatted at 2024-05-02 07:59:36 on dist-test-slave-f46z"
I20240502 07:59:42.415537 3892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:42.433826 3892 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:42.434612 3892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:42.434806 3892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:42.435389 3892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:42.436424 3892 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:42.436502 3892 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:42.436569 3892 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:42.436595 3892 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:42.445905 3892 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.133:40979
I20240502 07:59:42.445943 4019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.133:40979 every 8 connection(s)
I20240502 07:59:42.447034 3892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:59:42.451850 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 3892
I20240502 07:59:42.452296 18486 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskeLuhys/build/debug/bin/kudu
/tmp/dist-test-taskeLuhys/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.13.132:35129
--local_ip_for_outbound_sockets=127.18.13.132
--tserver_master_addrs=127.18.13.190:45049
--webserver_port=42721
--webserver_interface=127.18.13.132
--builtin_ntp_servers=127.18.13.148:37855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 07:59:42.457051 4020 heartbeater.cc:347] Connected to a master server at 127.18.13.190:45049
I20240502 07:59:42.457320 4020 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:42.457944 4020 heartbeater.cc:511] Master 127.18.13.190:45049 requested a full tablet report, sending...
I20240502 07:59:42.459218 2930 ts_manager.cc:194] Re-registered known tserver with Master: 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979)
I20240502 07:59:42.459864 2930 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.133:38885
W20240502 07:59:42.568696 4023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:42.568954 4023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:42.569025 4023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:42.572327 4023 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:42.572425 4023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:42.572531 4023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.13.132
I20240502 07:59:42.576741 4023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.13.148:37855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.13.132:35129
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.18.13.132
--webserver_port=42721
--tserver_master_addrs=127.18.13.190:45049
--never_fsync=true
--heap_profile_path=/tmp/kudu.4023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.13.132
--log_dir=/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:42.577795 4023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:42.579010 4023 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:42.588706 4031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:42.589447 4028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:42.589883 4029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:42.590564 4023 server_base.cc:1034] running on GCE node
I20240502 07:59:42.591037 4023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:42.591708 4023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:42.592875 4023 hybrid_clock.cc:648] HybridClock initialized: now 1714636782592860 us; error 35 us; skew 500 ppm
I20240502 07:59:42.595031 4023 webserver.cc:460] Webserver started at http://127.18.13.132:42721/ using document root <none> and password file <none>
I20240502 07:59:42.595654 4023 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:42.595738 4023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:42.599723 4023 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 07:59:42.601835 4037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:42.602924 4023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 07:59:42.603065 4023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "73bb095f4df448ae81da20d1a671547c"
format_stamp: "Formatted at 2024-05-02 07:59:36 on dist-test-slave-f46z"
I20240502 07:59:42.603513 4023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:42.625243 4023 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:42.625946 4023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:42.626123 4023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:42.626718 4023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:42.627683 4023 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:42.627734 4023 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:42.627817 4023 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:42.627849 4023 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:42.636945 4023 rpc_server.cc:307] RPC server started. Bound to: 127.18.13.132:35129
I20240502 07:59:42.636962 4150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.13.132:35129 every 8 connection(s)
I20240502 07:59:42.638186 4023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:59:42.639189 18486 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskeLuhys/build/debug/bin/kudu as pid 4023
I20240502 07:59:42.648927 4151 heartbeater.cc:347] Connected to a master server at 127.18.13.190:45049
I20240502 07:59:42.649255 4151 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:42.650126 4151 heartbeater.cc:511] Master 127.18.13.190:45049 requested a full tablet report, sending...
I20240502 07:59:42.651491 2930 ts_manager.cc:194] Re-registered known tserver with Master: 73bb095f4df448ae81da20d1a671547c (127.18.13.132:35129)
I20240502 07:59:42.652390 2930 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.13.132:59341
I20240502 07:59:42.861030 3188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:42.861507 3954 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:42.862969 3057 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:42.864307 4085 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:42.870522 3320 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:42.971056 4085 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:42.972440 3954 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:42.974900 3320 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:42.983480 3188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:42.985198 3057 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:43.110543 3954 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:43.113427 4085 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:43.113957 3188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:43.121724 3057 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:43.122519 3320 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:43.207891 3954 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:43.207942 3188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:43.210322 4085 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:43.211400 3057 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:43.222115 3320 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:43.255286 3341 consensus_queue.cc:237] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:43.257006 3210 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:43.257155 3076 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:43.257489 4300 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:43.257710 3832 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:43.260092 4300 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:43.260807 3210 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
W20240502 07:59:43.261487 3277 consensus_peers.cc:485] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 -> Peer 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979): Couldn't send request to peer 2a45c4829a5d41c4bc1c1e9c91277264. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8229b8193e454089b8e7e72973fe5e65. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:43.262552 3076 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:43.264396 2930 catalog_manager.cc:5484] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: config changed from index -1 to 4, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:43.273232 3210 consensus_queue.cc:237] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:43.274909 3341 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:43.275206 3076 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:43.275540 3726 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:43.275947 4315 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:43.277570 4314 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:43.278040 3341 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:43.280196 3076 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:43.280274 2929 catalog_manager.cc:5484] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: config changed from index -1 to 4, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:43.282047 3144 consensus_peers.cc:485] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c -> Peer 73bb095f4df448ae81da20d1a671547c (127.18.13.132:35129): Couldn't send request to peer 73bb095f4df448ae81da20d1a671547c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3429dee9c9e24927992401f2af945375. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:43.289168 3341 consensus_queue.cc:237] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:43.290766 3210 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:43.290807 3076 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:59:43.290849 3277 consensus_peers.cc:485] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 -> Peer 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979): Couldn't send request to peer 2a45c4829a5d41c4bc1c1e9c91277264. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f0ea65032f64a6fb74b88fb4fc92f0b. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:43.291203 4300 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:43.291777 4300 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:43.293121 4299 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:43.293545 3076 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:43.295599 3210 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:43.295945 2930 catalog_manager.cc:5484] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index -1 to 4, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:43.303857 3210 consensus_queue.cc:237] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:43.305292 3341 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:43.305322 3076 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:43.305708 4315 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:43.305914 3726 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:59:43.306104 3144 consensus_peers.cc:485] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c -> Peer 73bb095f4df448ae81da20d1a671547c (127.18.13.132:35129): Couldn't send request to peer 73bb095f4df448ae81da20d1a671547c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6b4e558aad64651aad857a4cbfac9d9. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:43.307839 4314 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:43.308310 3341 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:43.310029 3076 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:43.310410 2928 catalog_manager.cc:5484] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index -1 to 4, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:43.319777 3076 consensus_queue.cc:237] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:43.321560 3210 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:43.321653 3341 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:43.322129 3891 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:43.322367 4341 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:43.324447 4341 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:43.324788 3210 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:43.324959 3341 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
W20240502 07:59:43.325472 3013 consensus_peers.cc:485] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 -> Peer 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979): Couldn't send request to peer 2a45c4829a5d41c4bc1c1e9c91277264. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cdfb5d62286445b98e04d397fa8ffdb1. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:43.327350 2929 catalog_manager.cc:5484] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 reported cstate change: config changed from index -1 to 4, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "d11d320786dd4f1094ce083e43bfd5b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:43.334276 3341 consensus_queue.cc:237] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:43.335774 3076 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:43.336300 3210 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:43.336477 3832 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:43.336706 4300 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:43.338970 3832 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:43.340452 3210 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:43.341132 3076 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:43.341601 2928 catalog_manager.cc:5484] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index -1 to 4, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:43.344676 3276 consensus_peers.cc:485] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 -> Peer 73bb095f4df448ae81da20d1a671547c (127.18.13.132:35129): Couldn't send request to peer 73bb095f4df448ae81da20d1a671547c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e10362e44ff490da447b508aedf4226. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:43.461990 4020 heartbeater.cc:503] Master 127.18.13.190:45049 was elected leader, sending a full tablet report...
I20240502 07:59:43.655217 4151 heartbeater.cc:503] Master 127.18.13.190:45049 was elected leader, sending a full tablet report...
I20240502 07:59:43.738507 4415 ts_tablet_manager.cc:902] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c: Initiating tablet copy from peer e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:43.739252 4415 tablet_copy_client.cc:308] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:43705
I20240502 07:59:43.743041 3362 tablet_copy_service.cc:140] P e51a15732ba5427cbe7cf7466dbf9274: Received BeginTabletCopySession request for tablet 0e10362e44ff490da447b508aedf4226 from peer 73bb095f4df448ae81da20d1a671547c ({username='slave'} at 127.18.13.132:39575)
I20240502 07:59:43.743202 3362 tablet_copy_service.cc:161] P e51a15732ba5427cbe7cf7466dbf9274: Beginning new tablet copy session on tablet 0e10362e44ff490da447b508aedf4226 from peer 73bb095f4df448ae81da20d1a671547c at {username='slave'} at 127.18.13.132:39575: session id = 73bb095f4df448ae81da20d1a671547c-0e10362e44ff490da447b508aedf4226
I20240502 07:59:43.744488 3362 tablet_copy_source_session.cc:205] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:43.744690 4421 ts_tablet_manager.cc:902] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c: Initiating tablet copy from peer 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:43.745258 4421 tablet_copy_client.cc:308] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:42851
I20240502 07:59:43.748121 4415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e10362e44ff490da447b508aedf4226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.749354 3230 tablet_copy_service.cc:140] P 3dc087b5db164c89bea2988820a9b31c: Received BeginTabletCopySession request for tablet 3429dee9c9e24927992401f2af945375 from peer 73bb095f4df448ae81da20d1a671547c ({username='slave'} at 127.18.13.132:34233)
I20240502 07:59:43.749506 3230 tablet_copy_service.cc:161] P 3dc087b5db164c89bea2988820a9b31c: Beginning new tablet copy session on tablet 3429dee9c9e24927992401f2af945375 from peer 73bb095f4df448ae81da20d1a671547c at {username='slave'} at 127.18.13.132:34233: session id = 73bb095f4df448ae81da20d1a671547c-3429dee9c9e24927992401f2af945375
I20240502 07:59:43.751080 3230 tablet_copy_source_session.cc:205] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:43.751965 4426 ts_tablet_manager.cc:902] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264: Initiating tablet copy from peer e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:43.752297 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3429dee9c9e24927992401f2af945375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.752779 4426 tablet_copy_client.cc:308] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:43705
I20240502 07:59:43.755250 4421 tablet_copy_client.cc:783] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:43.755311 4415 tablet_copy_client.cc:783] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:43.755438 4415 tablet_copy_client.cc:647] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:43.755465 4421 tablet_copy_client.cc:647] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:43.757366 4421 tablet_copy_client.cc:523] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:43.757359 3361 tablet_copy_service.cc:140] P e51a15732ba5427cbe7cf7466dbf9274: Received BeginTabletCopySession request for tablet 8229b8193e454089b8e7e72973fe5e65 from peer 2a45c4829a5d41c4bc1c1e9c91277264 ({username='slave'} at 127.18.13.133:40463)
I20240502 07:59:43.757442 3361 tablet_copy_service.cc:161] P e51a15732ba5427cbe7cf7466dbf9274: Beginning new tablet copy session on tablet 8229b8193e454089b8e7e72973fe5e65 from peer 2a45c4829a5d41c4bc1c1e9c91277264 at {username='slave'} at 127.18.13.133:40463: session id = 2a45c4829a5d41c4bc1c1e9c91277264-8229b8193e454089b8e7e72973fe5e65
I20240502 07:59:43.758217 3361 tablet_copy_source_session.cc:205] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:43.758808 4415 tablet_copy_client.cc:523] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:43.759115 4421 tablet_bootstrap.cc:492] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c: Bootstrap starting.
I20240502 07:59:43.759816 4415 tablet_bootstrap.cc:492] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c: Bootstrap starting.
I20240502 07:59:43.761556 4426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8229b8193e454089b8e7e72973fe5e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.764673 4415 log.cc:824] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:43.768388 4426 tablet_copy_client.cc:783] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:43.768605 4426 tablet_copy_client.cc:647] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:43.770274 4426 tablet_copy_client.cc:523] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:43.772228 4426 tablet_bootstrap.cc:492] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap starting.
I20240502 07:59:43.779568 4426 log.cc:824] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:43.784865 4415 tablet_bootstrap.cc:492] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:43.785272 4415 tablet_bootstrap.cc:492] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c: Bootstrap complete.
I20240502 07:59:43.785629 4415 ts_tablet_manager.cc:1367] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c: Time spent bootstrapping tablet: real 0.026s user 0.026s sys 0.000s
I20240502 07:59:43.787961 4415 raft_consensus.cc:348] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:43.788179 4415 raft_consensus.cc:729] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 73bb095f4df448ae81da20d1a671547c, State: Initialized, Role: LEARNER
I20240502 07:59:43.788523 4415 consensus_queue.cc:260] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:43.789418 4415 ts_tablet_manager.cc:1398] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 07:59:43.790371 3361 tablet_copy_service.cc:342] P e51a15732ba5427cbe7cf7466dbf9274: Request end of tablet copy session 73bb095f4df448ae81da20d1a671547c-0e10362e44ff490da447b508aedf4226 received from {username='slave'} at 127.18.13.132:39575
I20240502 07:59:43.790474 3361 tablet_copy_service.cc:433] P e51a15732ba5427cbe7cf7466dbf9274: ending tablet copy session 73bb095f4df448ae81da20d1a671547c-0e10362e44ff490da447b508aedf4226 on tablet 0e10362e44ff490da447b508aedf4226 with peer 73bb095f4df448ae81da20d1a671547c
I20240502 07:59:43.790539 4421 tablet_bootstrap.cc:492] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:43.790825 4421 tablet_bootstrap.cc:492] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c: Bootstrap complete.
I20240502 07:59:43.791003 4421 ts_tablet_manager.cc:1367] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c: Time spent bootstrapping tablet: real 0.032s user 0.025s sys 0.004s
I20240502 07:59:43.792069 4421 raft_consensus.cc:348] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:43.792264 4421 raft_consensus.cc:729] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 73bb095f4df448ae81da20d1a671547c, State: Initialized, Role: LEARNER
I20240502 07:59:43.792407 4421 consensus_queue.cc:260] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:43.792692 4421 ts_tablet_manager.cc:1398] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:43.793534 3230 tablet_copy_service.cc:342] P 3dc087b5db164c89bea2988820a9b31c: Request end of tablet copy session 73bb095f4df448ae81da20d1a671547c-3429dee9c9e24927992401f2af945375 received from {username='slave'} at 127.18.13.132:34233
I20240502 07:59:43.793638 3230 tablet_copy_service.cc:433] P 3dc087b5db164c89bea2988820a9b31c: ending tablet copy session 73bb095f4df448ae81da20d1a671547c-3429dee9c9e24927992401f2af945375 on tablet 3429dee9c9e24927992401f2af945375 with peer 73bb095f4df448ae81da20d1a671547c
I20240502 07:59:43.803695 4426 tablet_bootstrap.cc:492] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:43.804109 4426 tablet_bootstrap.cc:492] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap complete.
I20240502 07:59:43.804419 4426 ts_tablet_manager.cc:1367] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent bootstrapping tablet: real 0.032s user 0.031s sys 0.000s
I20240502 07:59:43.806416 4426 raft_consensus.cc:348] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:43.806661 4426 raft_consensus.cc:729] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a45c4829a5d41c4bc1c1e9c91277264, State: Initialized, Role: LEARNER
I20240502 07:59:43.807116 4426 consensus_queue.cc:260] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:43.808279 4426 ts_tablet_manager.cc:1398] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 07:59:43.809232 3361 tablet_copy_service.cc:342] P e51a15732ba5427cbe7cf7466dbf9274: Request end of tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-8229b8193e454089b8e7e72973fe5e65 received from {username='slave'} at 127.18.13.133:40463
I20240502 07:59:43.809334 3361 tablet_copy_service.cc:433] P e51a15732ba5427cbe7cf7466dbf9274: ending tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-8229b8193e454089b8e7e72973fe5e65 on tablet 8229b8193e454089b8e7e72973fe5e65 with peer 2a45c4829a5d41c4bc1c1e9c91277264
I20240502 07:59:43.848557 4426 ts_tablet_manager.cc:902] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264: Initiating tablet copy from peer d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063)
I20240502 07:59:43.849005 4426 tablet_copy_client.cc:308] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37063
I20240502 07:59:43.852435 3098 tablet_copy_service.cc:140] P d11d320786dd4f1094ce083e43bfd5b5: Received BeginTabletCopySession request for tablet cdfb5d62286445b98e04d397fa8ffdb1 from peer 2a45c4829a5d41c4bc1c1e9c91277264 ({username='slave'} at 127.18.13.133:42873)
I20240502 07:59:43.852598 3098 tablet_copy_service.cc:161] P d11d320786dd4f1094ce083e43bfd5b5: Beginning new tablet copy session on tablet cdfb5d62286445b98e04d397fa8ffdb1 from peer 2a45c4829a5d41c4bc1c1e9c91277264 at {username='slave'} at 127.18.13.133:42873: session id = 2a45c4829a5d41c4bc1c1e9c91277264-cdfb5d62286445b98e04d397fa8ffdb1
I20240502 07:59:43.853981 3098 tablet_copy_source_session.cc:205] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:43.855075 4426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfb5d62286445b98e04d397fa8ffdb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.856956 4426 tablet_copy_client.cc:783] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:43.857095 4426 tablet_copy_client.cc:647] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:43.858534 4426 tablet_copy_client.cc:523] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:43.859599 4426 tablet_bootstrap.cc:492] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap starting.
I20240502 07:59:43.877296 4426 tablet_bootstrap.cc:492] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:43.877583 4426 tablet_bootstrap.cc:492] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap complete.
I20240502 07:59:43.877746 4426 ts_tablet_manager.cc:1367] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20240502 07:59:43.878278 4426 raft_consensus.cc:348] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:43.878397 4426 raft_consensus.cc:729] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a45c4829a5d41c4bc1c1e9c91277264, State: Initialized, Role: LEARNER
I20240502 07:59:43.878484 4426 consensus_queue.cc:260] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:43.878749 4426 ts_tablet_manager.cc:1398] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:43.879489 3098 tablet_copy_service.cc:342] P d11d320786dd4f1094ce083e43bfd5b5: Request end of tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-cdfb5d62286445b98e04d397fa8ffdb1 received from {username='slave'} at 127.18.13.133:42873
I20240502 07:59:43.879607 3098 tablet_copy_service.cc:433] P d11d320786dd4f1094ce083e43bfd5b5: ending tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-cdfb5d62286445b98e04d397fa8ffdb1 on tablet cdfb5d62286445b98e04d397fa8ffdb1 with peer 2a45c4829a5d41c4bc1c1e9c91277264
I20240502 07:59:43.908613 4426 ts_tablet_manager.cc:902] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264: Initiating tablet copy from peer e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:43.909173 4426 tablet_copy_client.cc:308] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:43705
I20240502 07:59:43.909689 3361 tablet_copy_service.cc:140] P e51a15732ba5427cbe7cf7466dbf9274: Received BeginTabletCopySession request for tablet 7f0ea65032f64a6fb74b88fb4fc92f0b from peer 2a45c4829a5d41c4bc1c1e9c91277264 ({username='slave'} at 127.18.13.133:40463)
I20240502 07:59:43.909816 3361 tablet_copy_service.cc:161] P e51a15732ba5427cbe7cf7466dbf9274: Beginning new tablet copy session on tablet 7f0ea65032f64a6fb74b88fb4fc92f0b from peer 2a45c4829a5d41c4bc1c1e9c91277264 at {username='slave'} at 127.18.13.133:40463: session id = 2a45c4829a5d41c4bc1c1e9c91277264-7f0ea65032f64a6fb74b88fb4fc92f0b
I20240502 07:59:43.910533 3361 tablet_copy_source_session.cc:205] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:43.911376 4426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0ea65032f64a6fb74b88fb4fc92f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.913152 4426 tablet_copy_client.cc:783] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:43.913301 4426 tablet_copy_client.cc:647] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:43.914417 4426 tablet_copy_client.cc:523] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:43.915442 4426 tablet_bootstrap.cc:492] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap starting.
I20240502 07:59:43.923565 4421 ts_tablet_manager.cc:902] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c: Initiating tablet copy from peer 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:43.924131 4421 tablet_copy_client.cc:308] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:42851
I20240502 07:59:43.924716 3230 tablet_copy_service.cc:140] P 3dc087b5db164c89bea2988820a9b31c: Received BeginTabletCopySession request for tablet e6b4e558aad64651aad857a4cbfac9d9 from peer 73bb095f4df448ae81da20d1a671547c ({username='slave'} at 127.18.13.132:34233)
I20240502 07:59:43.924834 3230 tablet_copy_service.cc:161] P 3dc087b5db164c89bea2988820a9b31c: Beginning new tablet copy session on tablet e6b4e558aad64651aad857a4cbfac9d9 from peer 73bb095f4df448ae81da20d1a671547c at {username='slave'} at 127.18.13.132:34233: session id = 73bb095f4df448ae81da20d1a671547c-e6b4e558aad64651aad857a4cbfac9d9
I20240502 07:59:43.925594 3230 tablet_copy_source_session.cc:205] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:43.926522 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b4e558aad64651aad857a4cbfac9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:43.928480 4421 tablet_copy_client.cc:783] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:43.928642 4421 tablet_copy_client.cc:647] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:43.929841 4421 tablet_copy_client.cc:523] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:43.930979 4421 tablet_bootstrap.cc:492] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c: Bootstrap starting.
I20240502 07:59:43.933466 4426 tablet_bootstrap.cc:492] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:43.933790 4426 tablet_bootstrap.cc:492] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap complete.
I20240502 07:59:43.933981 4426 ts_tablet_manager.cc:1367] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20240502 07:59:43.934406 4426 raft_consensus.cc:348] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:43.934521 4426 raft_consensus.cc:729] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a45c4829a5d41c4bc1c1e9c91277264, State: Initialized, Role: LEARNER
I20240502 07:59:43.934625 4426 consensus_queue.cc:260] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:43.934892 4426 ts_tablet_manager.cc:1398] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:43.935534 3361 tablet_copy_service.cc:342] P e51a15732ba5427cbe7cf7466dbf9274: Request end of tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-7f0ea65032f64a6fb74b88fb4fc92f0b received from {username='slave'} at 127.18.13.133:40463
I20240502 07:59:43.935618 3361 tablet_copy_service.cc:433] P e51a15732ba5427cbe7cf7466dbf9274: ending tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-7f0ea65032f64a6fb74b88fb4fc92f0b on tablet 7f0ea65032f64a6fb74b88fb4fc92f0b with peer 2a45c4829a5d41c4bc1c1e9c91277264
I20240502 07:59:43.950470 4421 tablet_bootstrap.cc:492] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:43.950775 4421 tablet_bootstrap.cc:492] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c: Bootstrap complete.
I20240502 07:59:43.950959 4421 ts_tablet_manager.cc:1367] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.004s
I20240502 07:59:43.951376 4421 raft_consensus.cc:348] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:43.951507 4421 raft_consensus.cc:729] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 73bb095f4df448ae81da20d1a671547c, State: Initialized, Role: LEARNER
I20240502 07:59:43.951625 4421 consensus_queue.cc:260] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:43.951912 4421 ts_tablet_manager.cc:1398] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 07:59:43.952598 3230 tablet_copy_service.cc:342] P 3dc087b5db164c89bea2988820a9b31c: Request end of tablet copy session 73bb095f4df448ae81da20d1a671547c-e6b4e558aad64651aad857a4cbfac9d9 received from {username='slave'} at 127.18.13.132:34233
I20240502 07:59:43.952692 3230 tablet_copy_service.cc:433] P 3dc087b5db164c89bea2988820a9b31c: ending tablet copy session 73bb095f4df448ae81da20d1a671547c-e6b4e558aad64651aad857a4cbfac9d9 on tablet e6b4e558aad64651aad857a4cbfac9d9 with peer 73bb095f4df448ae81da20d1a671547c
I20240502 07:59:44.151417 4105 raft_consensus.cc:1203] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:44.252133 3974 raft_consensus.cc:1203] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:44.271580 3974 raft_consensus.cc:1203] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:44.307062 4105 raft_consensus.cc:1203] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:44.328187 3974 raft_consensus.cc:1203] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:44.412437 4105 raft_consensus.cc:1203] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:59:44.665678 4517 raft_consensus.cc:1050] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274: attempting to promote NON_VOTER 73bb095f4df448ae81da20d1a671547c to VOTER
I20240502 07:59:44.666179 4517 consensus_queue.cc:237] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:44.667841 4105 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.667910 3078 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.667939 3210 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.668359 4514 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.668577 4300 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.668846 4513 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.670884 4513 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.671407 3078 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.672255 4105 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.672803 3210 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.674365 2929 catalog_manager.cc:5484] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index 4 to 5, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:44.678304 3341 consensus_queue.cc:237] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:44.679458 3210 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:44.679455 4105 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:44.679873 4517 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:44.680056 4514 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:44.681689 4300 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.682220 3210 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.682250 4105 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.683743 2923 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e10362e44ff490da447b508aedf4226 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:44.684688 2930 catalog_manager.cc:5484] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index 5 to 6, VOTER d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:44.687377 3057 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e10362e44ff490da447b508aedf4226 with delete_type TABLET_DATA_TOMBSTONED (TS d11d320786dd4f1094ce083e43bfd5b5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52530
I20240502 07:59:44.687958 4588 tablet_replica.cc:330] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5: stopping tablet replica
I20240502 07:59:44.688272 4588 raft_consensus.cc:2229] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:44.688474 4588 raft_consensus.cc:2258] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:44.689487 4588 ts_tablet_manager.cc:1875] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:44.691365 4588 ts_tablet_manager.cc:1888] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:44.691442 4588 log.cc:1196] T 0e10362e44ff490da447b508aedf4226 P d11d320786dd4f1094ce083e43bfd5b5: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/0e10362e44ff490da447b508aedf4226
I20240502 07:59:44.692070 2921 catalog_manager.cc:4830] TS d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063): tablet 0e10362e44ff490da447b508aedf4226 (table rebalance_test_table_0 [id=38b61401424d44728fb0d5830c0766fb]) successfully deleted
I20240502 07:59:44.761512 4594 raft_consensus.cc:1050] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5: attempting to promote NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 to VOTER
I20240502 07:59:44.762136 4594 consensus_queue.cc:237] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:44.763805 3341 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.763811 3974 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.763825 3210 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.764278 4594 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.764588 4597 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.764829 4596 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.767063 4594 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.767541 3341 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.769398 3974 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.769711 3210 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.770599 2933 catalog_manager.cc:5484] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 reported cstate change: config changed from index 4 to 5, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d11d320786dd4f1094ce083e43bfd5b5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:44.774405 3078 consensus_queue.cc:237] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:44.775831 3974 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:44.776297 4606 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:44.775947 3341 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:44.776829 4606 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:44.778139 4516 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.780071 2921 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cdfb5d62286445b98e04d397fa8ffdb1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:44.780640 2929 catalog_manager.cc:5484] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 reported cstate change: config changed from index 5 to 6, VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "d11d320786dd4f1094ce083e43bfd5b5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:44.780953 3341 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.781554 4517 raft_consensus.cc:1050] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274: attempting to promote NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 to VOTER
I20240502 07:59:44.781924 4517 consensus_queue.cc:237] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:44.783855 3188 tablet_service.cc:1514] Processing DeleteTablet for tablet cdfb5d62286445b98e04d397fa8ffdb1 with delete_type TABLET_DATA_TOMBSTONED (TS 3dc087b5db164c89bea2988820a9b31c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46664
I20240502 07:59:44.784365 4613 tablet_replica.cc:330] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c: stopping tablet replica
I20240502 07:59:44.784758 4613 raft_consensus.cc:2229] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:44.784821 3973 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.785058 4613 raft_consensus.cc:2258] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:44.786201 4300 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.786325 4613 ts_tablet_manager.cc:1875] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:44.788506 3210 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.788558 3078 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.789104 4300 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.789664 4300 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.789897 4513 raft_consensus.cc:1050] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274: attempting to promote NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 to VOTER
I20240502 07:59:44.790277 4513 consensus_queue.cc:237] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:44.791118 4538 raft_consensus.cc:1050] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c: attempting to promote NON_VOTER 73bb095f4df448ae81da20d1a671547c to VOTER
I20240502 07:59:44.791646 4538 consensus_queue.cc:237] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:44.791846 3078 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.791940 3209 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.792181 3973 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.792495 4514 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.793365 3341 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.793507 3076 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.793339 4514 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.793840 4300 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.794278 4314 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.794939 4105 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.795194 3974 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.796175 4314 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.796795 4517 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.798965 4539 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.799086 3972 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.800168 3209 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.801055 3078 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.801189 2930 catalog_manager.cc:5484] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index 4 to 5, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:44.802547 4514 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.804729 3341 consensus_queue.cc:237] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:44.804929 2933 catalog_manager.cc:5484] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: config changed from index 4 to 5, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.805583 4538 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.806124 3076 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.806592 3972 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:44.807029 4513 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:44.807613 3078 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:44.807718 3341 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.808163 4630 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:44.809051 2933 catalog_manager.cc:5484] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5 reported cstate change: config changed from index 4 to 5, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.810071 4517 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.810432 3078 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.810950 4613 ts_tablet_manager.cc:1888] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:44.811044 4613 log.cc:1196] T cdfb5d62286445b98e04d397fa8ffdb1 P 3dc087b5db164c89bea2988820a9b31c: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/cdfb5d62286445b98e04d397fa8ffdb1
I20240502 07:59:44.811407 3076 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.811643 2923 catalog_manager.cc:4830] TS 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851): tablet cdfb5d62286445b98e04d397fa8ffdb1 (table rebalance_test_table_1 [id=90a1f5d24e6f413eb4ef048ae0e4d46e]) successfully deleted
I20240502 07:59:44.811992 2923 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7f0ea65032f64a6fb74b88fb4fc92f0b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:44.814908 3972 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.816367 2930 catalog_manager.cc:5484] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index 5 to 6, VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:44.816908 4105 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.817200 3973 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.819099 3188 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f0ea65032f64a6fb74b88fb4fc92f0b with delete_type TABLET_DATA_TOMBSTONED (TS 3dc087b5db164c89bea2988820a9b31c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46664
I20240502 07:59:44.819409 4613 tablet_replica.cc:330] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c: stopping tablet replica
I20240502 07:59:44.819550 4613 raft_consensus.cc:2229] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:44.819684 4613 pending_rounds.cc:62] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c: Trying to abort 1 pending ops.
I20240502 07:59:44.819765 4613 pending_rounds.cc:68] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023152270465814528 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "7f0ea65032f64a6fb74b88fb4fc92f0b" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } } }
I20240502 07:59:44.820699 4613 raft_consensus.cc:2894] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 07:59:44.820778 4613 raft_consensus.cc:2866] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 07:59:44.820830 4613 raft_consensus.cc:2258] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:44.821393 3341 consensus_queue.cc:237] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:44.822016 4613 ts_tablet_manager.cc:1875] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:44.823719 4613 ts_tablet_manager.cc:1888] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:44.824576 3076 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:44.824927 3973 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:44.824924 4513 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:44.824579 3210 consensus_queue.cc:237] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:44.825315 4513 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:44.826076 4613 log.cc:1196] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 3dc087b5db164c89bea2988820a9b31c: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/7f0ea65032f64a6fb74b88fb4fc92f0b
I20240502 07:59:44.826498 4105 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:44.826602 2923 catalog_manager.cc:4830] TS 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851): tablet 7f0ea65032f64a6fb74b88fb4fc92f0b (table rebalance_test_table_2 [id=242dd44a47a143d6ad62e53b54120b3a]) successfully deleted
I20240502 07:59:44.826870 4538 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:44.827009 4517 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.827389 3076 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.828002 3341 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:44.829386 2923 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8229b8193e454089b8e7e72973fe5e65 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:44.829375 4539 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:44.829305 4314 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.830003 2929 catalog_manager.cc:5484] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index 5 to 6, VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) evicted. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:44.830312 3341 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.830804 2923 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3429dee9c9e24927992401f2af945375 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:44.831861 2928 catalog_manager.cc:5484] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: config changed from index 5 to 6, VOTER d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:44.831969 4105 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.833468 3188 tablet_service.cc:1514] Processing DeleteTablet for tablet 8229b8193e454089b8e7e72973fe5e65 with delete_type TABLET_DATA_TOMBSTONED (TS 3dc087b5db164c89bea2988820a9b31c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46664
I20240502 07:59:44.833854 4613 tablet_replica.cc:330] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: stopping tablet replica
I20240502 07:59:44.834617 4613 raft_consensus.cc:2229] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:44.834827 4613 raft_consensus.cc:2258] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:44.835615 4613 ts_tablet_manager.cc:1875] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:44.835887 3057 tablet_service.cc:1514] Processing DeleteTablet for tablet 3429dee9c9e24927992401f2af945375 with delete_type TABLET_DATA_TOMBSTONED (TS d11d320786dd4f1094ce083e43bfd5b5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52530
I20240502 07:59:44.836181 4588 tablet_replica.cc:330] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5: stopping tablet replica
I20240502 07:59:44.836330 4588 raft_consensus.cc:2229] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:44.836498 4588 raft_consensus.cc:2258] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:44.836745 3973 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:44.837306 4588 ts_tablet_manager.cc:1875] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:44.839718 4588 ts_tablet_manager.cc:1888] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:44.839869 4588 log.cc:1196] T 3429dee9c9e24927992401f2af945375 P d11d320786dd4f1094ce083e43bfd5b5: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/3429dee9c9e24927992401f2af945375
I20240502 07:59:44.840416 2921 catalog_manager.cc:4830] TS d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063): tablet 3429dee9c9e24927992401f2af945375 (table rebalance_test_table_1 [id=90a1f5d24e6f413eb4ef048ae0e4d46e]) successfully deleted
I20240502 07:59:44.841887 4613 ts_tablet_manager.cc:1888] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:44.841953 4613 log.cc:1196] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-1/wal/wals/8229b8193e454089b8e7e72973fe5e65
I20240502 07:59:44.842394 2923 catalog_manager.cc:4830] TS 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851): tablet 8229b8193e454089b8e7e72973fe5e65 (table rebalance_test_table_0 [id=38b61401424d44728fb0d5830c0766fb]) successfully deleted
I20240502 07:59:44.929493 4314 raft_consensus.cc:1050] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c: attempting to promote NON_VOTER 73bb095f4df448ae81da20d1a671547c to VOTER
I20240502 07:59:44.929888 4314 consensus_queue.cc:237] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:44.931334 3341 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.931325 4105 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.931401 3076 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:44.931762 4539 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.931986 4538 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.932160 4314 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:44.933955 4539 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.934499 3076 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.934644 3341 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.936096 4105 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.938067 2929 catalog_manager.cc:5484] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5 reported cstate change: config changed from index 4 to 5, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } attrs { replace: true } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.941301 3210 consensus_queue.cc:237] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:44.942651 4105 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:44.943050 4314 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:44.943387 3341 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:44.943753 4314 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:44.944689 4619 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.945092 4105 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.945730 3341 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.946081 2923 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6b4e558aad64651aad857a4cbfac9d9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:44.947650 2928 catalog_manager.cc:5484] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c reported cstate change: config changed from index 5 to 6, VOTER d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129) evicted. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:44.950073 3057 tablet_service.cc:1514] Processing DeleteTablet for tablet e6b4e558aad64651aad857a4cbfac9d9 with delete_type TABLET_DATA_TOMBSTONED (TS d11d320786dd4f1094ce083e43bfd5b5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52530
I20240502 07:59:44.950345 4588 tablet_replica.cc:330] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5: stopping tablet replica
I20240502 07:59:44.950481 4588 raft_consensus.cc:2229] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:44.950630 4588 raft_consensus.cc:2258] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:44.951231 4588 ts_tablet_manager.cc:1875] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:44.952972 4588 ts_tablet_manager.cc:1888] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:59:44.953045 4588 log.cc:1196] T e6b4e558aad64651aad857a4cbfac9d9 P d11d320786dd4f1094ce083e43bfd5b5: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-0/wal/wals/e6b4e558aad64651aad857a4cbfac9d9
I20240502 07:59:44.953594 2921 catalog_manager.cc:4830] TS d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063): tablet e6b4e558aad64651aad857a4cbfac9d9 (table rebalance_test_table_2 [id=242dd44a47a143d6ad62e53b54120b3a]) successfully deleted
I20240502 07:59:45.128731 3057 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:45.130882 4085 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:45.131866 3954 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:45.132102 3188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:45.132644 3320 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:45.225229 3188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:45.227209 4085 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:45.227262 3954 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:45.237185 3057 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:45.238615 3320 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:45.325798 3954 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:45.327071 4085 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:45.327183 3188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:45.327490 3057 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:45.328294 3320 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:45.520327 2928 ts_manager.cc:292] Set tserver state for e51a15732ba5427cbe7cf7466dbf9274 to MAINTENANCE_MODE
I20240502 07:59:45.729446 3188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:45.730362 3057 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:45.731487 4085 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:45.731757 3954 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:45.732571 3320 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:45.832860 3188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:45.832891 3954 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:45.833380 4085 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:45.835675 3320 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:45.853186 3057 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:45.968920 4085 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:45.969236 3188 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:45.970265 3954 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:45.970847 3057 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:45.977481 3320 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:46.012075 3341 consensus_queue.cc:237] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:46.013330 3078 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:46.013527 3973 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:46.013741 4517 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:46.013991 4847 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:46.015486 4517 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:46.015926 3078 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:46.017573 3973 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:46.017920 2933 catalog_manager.cc:5484] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:46.021291 3341 consensus_queue.cc:237] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
W20240502 07:59:46.023135 3276 consensus_peers.cc:485] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 -> Peer 73bb095f4df448ae81da20d1a671547c (127.18.13.132:35129): Couldn't send request to peer 73bb095f4df448ae81da20d1a671547c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f0ea65032f64a6fb74b88fb4fc92f0b. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:46.023226 3078 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:46.023852 3973 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:46.024109 4889 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:46.024312 4847 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:46.026016 4847 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:46.026441 3973 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:46.028189 2923 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7f0ea65032f64a6fb74b88fb4fc92f0b with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:59:46.028807 2933 catalog_manager.cc:5484] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 reported cstate change: config changed from index 7 to 8, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:46.028791 3078 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:46.030231 3341 consensus_queue.cc:237] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } }
I20240502 07:59:46.031733 3210 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:46.032142 4889 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:46.032298 3078 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:46.033308 4889 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:46.034085 4517 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } }
I20240502 07:59:46.034847 3210 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } }
I20240502 07:59:46.036391 2928 catalog_manager.cc:5484] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240502 07:59:46.037472 3078 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } }
I20240502 07:59:46.040460 3341 consensus_queue.cc:237] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:46.041962 3210 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 07:59:46.042155 3277 consensus_peers.cc:485] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 -> Peer 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979): Couldn't send request to peer 2a45c4829a5d41c4bc1c1e9c91277264. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d73ac6edc2549e7a4c7dbc2bb91d750. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:46.042394 4517 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:46.042609 3078 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:46.043831 4517 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:46.044842 4889 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:46.045506 3209 consensus_queue.cc:237] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } }
I20240502 07:59:46.046118 2923 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:59:46.046557 3209 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:46.047089 3076 raft_consensus.cc:1261] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:46.048202 4314 consensus_queue.cc:1035] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:46.049017 3341 raft_consensus.cc:1261] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:46.049342 4904 consensus_queue.cc:1035] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:46.050470 2930 catalog_manager.cc:5484] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: config changed from index 4 to 5, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:46.050967 3076 raft_consensus.cc:2932] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } }
I20240502 07:59:46.050490 4905 raft_consensus.cc:2932] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } }
I20240502 07:59:46.051710 3078 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:46.051858 3341 raft_consensus.cc:2932] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } }
I20240502 07:59:46.056116 2929 catalog_manager.cc:5484] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } }
I20240502 07:59:46.059561 3209 consensus_queue.cc:237] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:46.061152 3341 raft_consensus.cc:1261] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:46.061215 3078 raft_consensus.cc:1261] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:46.061578 4314 consensus_queue.cc:1035] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:46.061795 4904 consensus_queue.cc:1035] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:46.063560 4314 raft_consensus.cc:2932] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:46.063871 3078 raft_consensus.cc:2932] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:46.065147 2923 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ee8e92d37ee14bfcbf0dd48ce054a339 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:59:46.065657 3341 raft_consensus.cc:2932] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:46.066375 2930 catalog_manager.cc:5484] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: config changed from index 4 to 5, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:59:46.066623 3145 consensus_peers.cc:485] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c -> Peer 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979): Couldn't send request to peer 2a45c4829a5d41c4bc1c1e9c91277264. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee8e92d37ee14bfcbf0dd48ce054a339. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:46.438494 4939 ts_tablet_manager.cc:902] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c: Initiating tablet copy from peer e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:46.439002 4939 tablet_copy_client.cc:308] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:43705
I20240502 07:59:46.439488 3361 tablet_copy_service.cc:140] P e51a15732ba5427cbe7cf7466dbf9274: Received BeginTabletCopySession request for tablet 7f0ea65032f64a6fb74b88fb4fc92f0b from peer 73bb095f4df448ae81da20d1a671547c ({username='slave'} at 127.18.13.132:39575)
I20240502 07:59:46.439591 3361 tablet_copy_service.cc:161] P e51a15732ba5427cbe7cf7466dbf9274: Beginning new tablet copy session on tablet 7f0ea65032f64a6fb74b88fb4fc92f0b from peer 73bb095f4df448ae81da20d1a671547c at {username='slave'} at 127.18.13.132:39575: session id = 73bb095f4df448ae81da20d1a671547c-7f0ea65032f64a6fb74b88fb4fc92f0b
I20240502 07:59:46.440402 3361 tablet_copy_source_session.cc:205] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:46.441305 4939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f0ea65032f64a6fb74b88fb4fc92f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:46.442760 4944 ts_tablet_manager.cc:902] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: Initiating tablet copy from peer e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:46.443404 4944 tablet_copy_client.cc:308] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:43705
I20240502 07:59:46.443552 4939 tablet_copy_client.cc:783] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:46.443725 4939 tablet_copy_client.cc:647] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:46.443902 3361 tablet_copy_service.cc:140] P e51a15732ba5427cbe7cf7466dbf9274: Received BeginTabletCopySession request for tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 from peer 2a45c4829a5d41c4bc1c1e9c91277264 ({username='slave'} at 127.18.13.133:40463)
I20240502 07:59:46.444011 3361 tablet_copy_service.cc:161] P e51a15732ba5427cbe7cf7466dbf9274: Beginning new tablet copy session on tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 from peer 2a45c4829a5d41c4bc1c1e9c91277264 at {username='slave'} at 127.18.13.133:40463: session id = 2a45c4829a5d41c4bc1c1e9c91277264-3d73ac6edc2549e7a4c7dbc2bb91d750
I20240502 07:59:46.444839 3361 tablet_copy_source_session.cc:205] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:46.444892 4939 tablet_copy_client.cc:523] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:46.445652 4944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d73ac6edc2549e7a4c7dbc2bb91d750. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:46.446028 4939 tablet_bootstrap.cc:492] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c: Bootstrap starting.
I20240502 07:59:46.447494 4944 tablet_copy_client.cc:783] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:46.447633 4944 tablet_copy_client.cc:647] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:46.454700 4944 tablet_copy_client.cc:523] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:46.455883 4944 tablet_bootstrap.cc:492] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap starting.
I20240502 07:59:46.467779 4939 tablet_bootstrap.cc:492] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:46.468101 4939 tablet_bootstrap.cc:492] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c: Bootstrap complete.
I20240502 07:59:46.468276 4939 ts_tablet_manager.cc:1367] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c: Time spent bootstrapping tablet: real 0.022s user 0.023s sys 0.000s
I20240502 07:59:46.468658 4939 raft_consensus.cc:348] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:46.468765 4939 raft_consensus.cc:729] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 73bb095f4df448ae81da20d1a671547c, State: Initialized, Role: LEARNER
I20240502 07:59:46.468863 4939 consensus_queue.cc:260] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:46.469199 4939 ts_tablet_manager.cc:1398] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:46.469741 3361 tablet_copy_service.cc:342] P e51a15732ba5427cbe7cf7466dbf9274: Request end of tablet copy session 73bb095f4df448ae81da20d1a671547c-7f0ea65032f64a6fb74b88fb4fc92f0b received from {username='slave'} at 127.18.13.132:39575
I20240502 07:59:46.469825 3361 tablet_copy_service.cc:433] P e51a15732ba5427cbe7cf7466dbf9274: ending tablet copy session 73bb095f4df448ae81da20d1a671547c-7f0ea65032f64a6fb74b88fb4fc92f0b on tablet 7f0ea65032f64a6fb74b88fb4fc92f0b with peer 73bb095f4df448ae81da20d1a671547c
I20240502 07:59:46.475430 4944 tablet_bootstrap.cc:492] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:46.475700 4944 tablet_bootstrap.cc:492] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap complete.
I20240502 07:59:46.475878 4944 ts_tablet_manager.cc:1367] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.004s
I20240502 07:59:46.476250 4944 raft_consensus.cc:348] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:46.476352 4944 raft_consensus.cc:729] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a45c4829a5d41c4bc1c1e9c91277264, State: Initialized, Role: LEARNER
I20240502 07:59:46.476429 4944 consensus_queue.cc:260] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:46.476668 4944 ts_tablet_manager.cc:1398] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:46.477177 3361 tablet_copy_service.cc:342] P e51a15732ba5427cbe7cf7466dbf9274: Request end of tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-3d73ac6edc2549e7a4c7dbc2bb91d750 received from {username='slave'} at 127.18.13.133:40463
I20240502 07:59:46.477270 3361 tablet_copy_service.cc:433] P e51a15732ba5427cbe7cf7466dbf9274: ending tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-3d73ac6edc2549e7a4c7dbc2bb91d750 on tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 with peer 2a45c4829a5d41c4bc1c1e9c91277264
I20240502 07:59:46.569658 4944 ts_tablet_manager.cc:902] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264: Initiating tablet copy from peer 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:46.570158 4944 tablet_copy_client.cc:308] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:42851
I20240502 07:59:46.573596 3230 tablet_copy_service.cc:140] P 3dc087b5db164c89bea2988820a9b31c: Received BeginTabletCopySession request for tablet ee8e92d37ee14bfcbf0dd48ce054a339 from peer 2a45c4829a5d41c4bc1c1e9c91277264 ({username='slave'} at 127.18.13.133:53805)
I20240502 07:59:46.573699 3230 tablet_copy_service.cc:161] P 3dc087b5db164c89bea2988820a9b31c: Beginning new tablet copy session on tablet ee8e92d37ee14bfcbf0dd48ce054a339 from peer 2a45c4829a5d41c4bc1c1e9c91277264 at {username='slave'} at 127.18.13.133:53805: session id = 2a45c4829a5d41c4bc1c1e9c91277264-ee8e92d37ee14bfcbf0dd48ce054a339
I20240502 07:59:46.574419 3230 tablet_copy_source_session.cc:205] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:46.575206 4944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee8e92d37ee14bfcbf0dd48ce054a339. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:46.576946 4944 tablet_copy_client.cc:783] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:46.577095 4944 tablet_copy_client.cc:647] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:46.578188 4944 tablet_copy_client.cc:523] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:46.579146 4944 tablet_bootstrap.cc:492] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap starting.
I20240502 07:59:46.595742 4944 tablet_bootstrap.cc:492] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:46.596076 4944 tablet_bootstrap.cc:492] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap complete.
I20240502 07:59:46.596261 4944 ts_tablet_manager.cc:1367] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent bootstrapping tablet: real 0.017s user 0.017s sys 0.000s
I20240502 07:59:46.596693 4944 raft_consensus.cc:348] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:46.596843 4944 raft_consensus.cc:729] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a45c4829a5d41c4bc1c1e9c91277264, State: Initialized, Role: LEARNER
I20240502 07:59:46.596962 4944 consensus_queue.cc:260] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:46.597280 4944 ts_tablet_manager.cc:1398] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:46.597819 3230 tablet_copy_service.cc:342] P 3dc087b5db164c89bea2988820a9b31c: Request end of tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-ee8e92d37ee14bfcbf0dd48ce054a339 received from {username='slave'} at 127.18.13.133:53805
I20240502 07:59:46.597951 3230 tablet_copy_service.cc:433] P 3dc087b5db164c89bea2988820a9b31c: ending tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-ee8e92d37ee14bfcbf0dd48ce054a339 on tablet ee8e92d37ee14bfcbf0dd48ce054a339 with peer 2a45c4829a5d41c4bc1c1e9c91277264
I20240502 07:59:46.950181 3973 raft_consensus.cc:1203] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:46.958593 4104 raft_consensus.cc:1203] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 07:59:47.007486 3973 raft_consensus.cc:1203] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:47.441601 5005 raft_consensus.cc:1050] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274: attempting to promote NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 to VOTER
I20240502 07:59:47.442081 5005 consensus_queue.cc:237] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:47.443686 3973 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:47.443809 3209 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:47.443809 3078 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:47.444200 5038 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:47.444401 5005 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:47.444550 4889 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:47.446705 4889 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:47.447209 3209 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:47.449340 3973 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:47.449617 2928 catalog_manager.cc:5484] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index 5 to 6, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:47.450176 3078 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:47.488328 5004 raft_consensus.cc:1050] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c: attempting to promote NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 to VOTER
I20240502 07:59:47.488796 5004 consensus_queue.cc:237] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:47.490343 3341 raft_consensus.cc:1261] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:47.490343 3078 raft_consensus.cc:1261] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:47.490388 3973 raft_consensus.cc:1261] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:47.490844 5048 consensus_queue.cc:1035] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:47.491114 5049 consensus_queue.cc:1035] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:47.491282 5046 consensus_queue.cc:1035] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:47.493436 5049 raft_consensus.cc:2932] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:47.493908 3341 raft_consensus.cc:2932] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:47.493939 3078 raft_consensus.cc:2932] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:47.495666 3973 raft_consensus.cc:2932] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:47.496640 2930 catalog_manager.cc:5484] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: config changed from index 5 to 6, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:47.500300 3209 consensus_queue.cc:237] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:47.501469 3973 raft_consensus.cc:1261] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:47.501523 3078 raft_consensus.cc:1261] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:47.501878 5048 consensus_queue.cc:1035] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:47.502081 5046 consensus_queue.cc:1035] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:47.503747 5046 raft_consensus.cc:2932] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:47.504246 3973 raft_consensus.cc:2932] T ee8e92d37ee14bfcbf0dd48ce054a339 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:47.504590 3078 raft_consensus.cc:2932] T ee8e92d37ee14bfcbf0dd48ce054a339 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:47.505422 2923 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ee8e92d37ee14bfcbf0dd48ce054a339 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:47.506740 2933 catalog_manager.cc:5484] T ee8e92d37ee14bfcbf0dd48ce054a339 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: config changed from index 6 to 7, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:47.509742 3320 tablet_service.cc:1514] Processing DeleteTablet for tablet ee8e92d37ee14bfcbf0dd48ce054a339 with delete_type TABLET_DATA_TOMBSTONED (TS e51a15732ba5427cbe7cf7466dbf9274 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53650
I20240502 07:59:47.510330 5053 tablet_replica.cc:330] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274: stopping tablet replica
I20240502 07:59:47.510637 5053 raft_consensus.cc:2229] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:47.510851 5053 raft_consensus.cc:2258] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:47.511982 5053 ts_tablet_manager.cc:1875] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:47.514081 5053 ts_tablet_manager.cc:1888] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 07:59:47.514170 5053 log.cc:1196] T ee8e92d37ee14bfcbf0dd48ce054a339 P e51a15732ba5427cbe7cf7466dbf9274: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/ee8e92d37ee14bfcbf0dd48ce054a339
I20240502 07:59:47.514786 2923 catalog_manager.cc:4830] TS e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705): tablet ee8e92d37ee14bfcbf0dd48ce054a339 (table rebalance_test_table_1 [id=90a1f5d24e6f413eb4ef048ae0e4d46e]) successfully deleted
I20240502 07:59:47.519172 4889 raft_consensus.cc:1050] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274: attempting to promote NON_VOTER 73bb095f4df448ae81da20d1a671547c to VOTER
I20240502 07:59:47.519560 4889 consensus_queue.cc:237] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:47.521165 3078 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:47.521157 4104 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:47.521337 3973 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:47.521647 5006 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:47.521858 4889 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:47.522025 5038 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:47.524019 5038 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:47.524513 4104 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:47.525009 3078 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:47.527284 2929 catalog_manager.cc:5484] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c reported cstate change: config changed from index 8 to 9, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:47.527452 3973 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:47.604904 3341 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7f0ea65032f64a6fb74b88fb4fc92f0b"
dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59384
I20240502 07:59:47.605119 3341 raft_consensus.cc:595] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:47.617095 3341 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d73ac6edc2549e7a4c7dbc2bb91d750"
dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59402
I20240502 07:59:47.617254 3341 raft_consensus.cc:595] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:47.825449 3209 consensus_queue.cc:237] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:47.826992 4104 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:47.827018 3341 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:47.827420 5048 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:47.827643 5049 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:47.829735 5046 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:47.830240 4104 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:47.833016 2928 catalog_manager.cc:5484] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:47.833431 5006 raft_consensus.cc:979] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274: : Instructing follower 2a45c4829a5d41c4bc1c1e9c91277264 to start an election
I20240502 07:59:47.833352 3341 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:47.833539 5006 raft_consensus.cc:1067] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Signalling peer 2a45c4829a5d41c4bc1c1e9c91277264 to start an election
I20240502 07:59:47.834635 3973 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d73ac6edc2549e7a4c7dbc2bb91d750"
dest_uuid: "2a45c4829a5d41c4bc1c1e9c91277264"
from {username='slave'} at 127.18.13.131:46759
I20240502 07:59:47.834833 3973 raft_consensus.cc:482] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:47.834973 3973 raft_consensus.cc:3037] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:47.835724 3973 raft_consensus.cc:504] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:47.837060 3973 leader_election.cc:288] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [CANDIDATE]: Term 2 election: Requested vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851), e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:47.837581 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d73ac6edc2549e7a4c7dbc2bb91d750" candidate_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5"
I20240502 07:59:47.837714 3078 raft_consensus.cc:3037] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:47.837713 3209 consensus_queue.cc:237] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:47.838793 3078 raft_consensus.cc:2454] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a45c4829a5d41c4bc1c1e9c91277264 in term 2.
I20240502 07:59:47.838933 3341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d73ac6edc2549e7a4c7dbc2bb91d750" candidate_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
I20240502 07:59:47.839047 3341 raft_consensus.cc:3032] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:47.839090 3341 raft_consensus.cc:729] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Running, Role: LEADER
I20240502 07:59:47.839246 3341 consensus_queue.cc:260] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:47.839306 4104 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:47.839463 3341 raft_consensus.cc:3037] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:47.839824 5048 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:47.840289 3341 raft_consensus.cc:2454] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a45c4829a5d41c4bc1c1e9c91277264 in term 2.
I20240502 07:59:47.840626 3910 leader_election.cc:302] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2a45c4829a5d41c4bc1c1e9c91277264, d11d320786dd4f1094ce083e43bfd5b5, e51a15732ba5427cbe7cf7466dbf9274; no voters:
I20240502 07:59:47.840926 4888 raft_consensus.cc:2781] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:47.841022 3209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d73ac6edc2549e7a4c7dbc2bb91d750" candidate_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "3dc087b5db164c89bea2988820a9b31c"
I20240502 07:59:47.841125 3209 raft_consensus.cc:3037] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:47.841125 4888 raft_consensus.cc:686] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 LEADER]: Becoming Leader. State: Replica: 2a45c4829a5d41c4bc1c1e9c91277264, State: Running, Role: LEADER
W20240502 07:59:47.841420 3145 consensus_peers.cc:485] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c -> Peer 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979): Couldn't send request to peer 2a45c4829a5d41c4bc1c1e9c91277264. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6b4e558aad64651aad857a4cbfac9d9. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:47.841465 4888 consensus_queue.cc:237] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:47.841781 3341 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:47.842096 3209 raft_consensus.cc:2454] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a45c4829a5d41c4bc1c1e9c91277264 in term 2.
I20240502 07:59:47.842630 5049 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:47.843057 4104 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:47.843940 5049 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:47.844321 2933 catalog_manager.cc:5484] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 reported cstate change: term changed from 1 to 2, leader changed from e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) to 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133). New cstate: current_term: 2 leader_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:47.844623 2923 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e6b4e558aad64651aad857a4cbfac9d9 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:59:47.845335 3341 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:47.845486 2930 catalog_manager.cc:5484] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c reported cstate change: config changed from index 7 to 8, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:47.858567 3341 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7f0ea65032f64a6fb74b88fb4fc92f0b"
dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59420
I20240502 07:59:47.858711 3341 raft_consensus.cc:595] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:47.906419 5038 raft_consensus.cc:979] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274: : Instructing follower 73bb095f4df448ae81da20d1a671547c to start an election
I20240502 07:59:47.906569 5038 raft_consensus.cc:1067] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Signalling peer 73bb095f4df448ae81da20d1a671547c to start an election
I20240502 07:59:47.907084 4104 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7f0ea65032f64a6fb74b88fb4fc92f0b"
dest_uuid: "73bb095f4df448ae81da20d1a671547c"
from {username='slave'} at 127.18.13.131:52365
I20240502 07:59:47.907279 4104 raft_consensus.cc:482] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:47.907397 4104 raft_consensus.cc:3037] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:47.908360 4104 raft_consensus.cc:504] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:47.909425 4104 leader_election.cc:288] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [CANDIDATE]: Term 2 election: Requested vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705), 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979)
I20240502 07:59:47.909788 3341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0ea65032f64a6fb74b88fb4fc92f0b" candidate_uuid: "73bb095f4df448ae81da20d1a671547c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
I20240502 07:59:47.909921 3341 raft_consensus.cc:3032] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:47.909963 3341 raft_consensus.cc:729] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Running, Role: LEADER
I20240502 07:59:47.910137 3341 consensus_queue.cc:260] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:47.910326 3341 raft_consensus.cc:3037] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:47.911384 3341 raft_consensus.cc:2454] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73bb095f4df448ae81da20d1a671547c in term 2.
I20240502 07:59:47.913858 3078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0ea65032f64a6fb74b88fb4fc92f0b" candidate_uuid: "73bb095f4df448ae81da20d1a671547c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5"
I20240502 07:59:47.914024 3078 raft_consensus.cc:3037] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:47.914095 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f0ea65032f64a6fb74b88fb4fc92f0b" candidate_uuid: "73bb095f4df448ae81da20d1a671547c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "2a45c4829a5d41c4bc1c1e9c91277264"
I20240502 07:59:47.914268 3973 raft_consensus.cc:3037] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:47.915109 3078 raft_consensus.cc:2454] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73bb095f4df448ae81da20d1a671547c in term 2.
I20240502 07:59:47.915248 3973 raft_consensus.cc:2454] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73bb095f4df448ae81da20d1a671547c in term 2.
I20240502 07:59:47.915498 4039 leader_election.cc:302] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 73bb095f4df448ae81da20d1a671547c, d11d320786dd4f1094ce083e43bfd5b5, e51a15732ba5427cbe7cf7466dbf9274; no voters:
I20240502 07:59:47.915990 5058 raft_consensus.cc:2781] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:47.916204 5058 raft_consensus.cc:686] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [term 2 LEADER]: Becoming Leader. State: Replica: 73bb095f4df448ae81da20d1a671547c, State: Running, Role: LEADER
I20240502 07:59:47.916509 5058 consensus_queue.cc:237] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:47.918980 2928 catalog_manager.cc:5484] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c reported cstate change: term changed from 1 to 2, leader changed from e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) to 73bb095f4df448ae81da20d1a671547c (127.18.13.132). New cstate: current_term: 2 leader_uuid: "73bb095f4df448ae81da20d1a671547c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:48.375054 3341 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Refusing update from remote peer 73bb095f4df448ae81da20d1a671547c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:59:48.375047 3078 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 2 FOLLOWER]: Refusing update from remote peer 73bb095f4df448ae81da20d1a671547c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:59:48.375548 5058 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:48.375751 5119 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:48.379086 3973 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Refusing update from remote peer 73bb095f4df448ae81da20d1a671547c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:59:48.379503 5152 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:48.414494 3341 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Refusing update from remote peer 2a45c4829a5d41c4bc1c1e9c91277264: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 07:59:48.415035 5155 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:48.426816 3209 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 2 FOLLOWER]: Refusing update from remote peer 2a45c4829a5d41c4bc1c1e9c91277264: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 07:59:48.427300 5155 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:48.430984 3078 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 2 FOLLOWER]: Refusing update from remote peer 2a45c4829a5d41c4bc1c1e9c91277264: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 07:59:48.431664 5158 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:48.464247 5160 ts_tablet_manager.cc:902] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264: Initiating tablet copy from peer 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:48.464787 5160 tablet_copy_client.cc:308] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:42851
I20240502 07:59:48.465349 3230 tablet_copy_service.cc:140] P 3dc087b5db164c89bea2988820a9b31c: Received BeginTabletCopySession request for tablet e6b4e558aad64651aad857a4cbfac9d9 from peer 2a45c4829a5d41c4bc1c1e9c91277264 ({username='slave'} at 127.18.13.133:53805)
I20240502 07:59:48.465498 3230 tablet_copy_service.cc:161] P 3dc087b5db164c89bea2988820a9b31c: Beginning new tablet copy session on tablet e6b4e558aad64651aad857a4cbfac9d9 from peer 2a45c4829a5d41c4bc1c1e9c91277264 at {username='slave'} at 127.18.13.133:53805: session id = 2a45c4829a5d41c4bc1c1e9c91277264-e6b4e558aad64651aad857a4cbfac9d9
I20240502 07:59:48.466214 3230 tablet_copy_source_session.cc:205] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:48.467118 5160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6b4e558aad64651aad857a4cbfac9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:48.469120 5160 tablet_copy_client.cc:783] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:48.469319 5160 tablet_copy_client.cc:647] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:48.470574 5160 tablet_copy_client.cc:523] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:48.471660 5160 tablet_bootstrap.cc:492] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap starting.
I20240502 07:59:48.491585 5160 tablet_bootstrap.cc:492] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:48.491946 5160 tablet_bootstrap.cc:492] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap complete.
I20240502 07:59:48.492113 5160 ts_tablet_manager.cc:1367] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent bootstrapping tablet: real 0.020s user 0.021s sys 0.000s
I20240502 07:59:48.492527 5160 raft_consensus.cc:348] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:48.492636 5160 raft_consensus.cc:729] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a45c4829a5d41c4bc1c1e9c91277264, State: Initialized, Role: LEARNER
I20240502 07:59:48.492722 5160 consensus_queue.cc:260] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:48.492990 5160 ts_tablet_manager.cc:1398] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:48.493652 3230 tablet_copy_service.cc:342] P 3dc087b5db164c89bea2988820a9b31c: Request end of tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-e6b4e558aad64651aad857a4cbfac9d9 received from {username='slave'} at 127.18.13.133:53805
I20240502 07:59:48.493749 3230 tablet_copy_service.cc:433] P 3dc087b5db164c89bea2988820a9b31c: ending tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-e6b4e558aad64651aad857a4cbfac9d9 on tablet e6b4e558aad64651aad857a4cbfac9d9 with peer 2a45c4829a5d41c4bc1c1e9c91277264
I20240502 07:59:48.780063 4104 consensus_queue.cc:237] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:48.781309 3078 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 2 FOLLOWER]: Refusing update from remote peer 73bb095f4df448ae81da20d1a671547c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 07:59:48.781699 3973 raft_consensus.cc:1261] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Refusing update from remote peer 73bb095f4df448ae81da20d1a671547c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 07:59:48.781744 5058 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:48.782056 5152 consensus_queue.cc:1035] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:48.783591 5058 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:48.784415 3973 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:48.785889 2921 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7f0ea65032f64a6fb74b88fb4fc92f0b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:48.786080 2928 catalog_manager.cc:5484] T 7f0ea65032f64a6fb74b88fb4fc92f0b P 73bb095f4df448ae81da20d1a671547c reported cstate change: config changed from index 9 to 11, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New cstate: current_term: 2 leader_uuid: "73bb095f4df448ae81da20d1a671547c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:48.786286 3078 raft_consensus.cc:2932] T 7f0ea65032f64a6fb74b88fb4fc92f0b P d11d320786dd4f1094ce083e43bfd5b5 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:48.789273 3320 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f0ea65032f64a6fb74b88fb4fc92f0b with delete_type TABLET_DATA_TOMBSTONED (TS e51a15732ba5427cbe7cf7466dbf9274 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:53650
I20240502 07:59:48.789731 5200 tablet_replica.cc:330] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274: stopping tablet replica
I20240502 07:59:48.789886 5200 raft_consensus.cc:2229] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:48.790032 5200 raft_consensus.cc:2258] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:48.790625 5200 ts_tablet_manager.cc:1875] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:48.792675 5200 ts_tablet_manager.cc:1888] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 07:59:48.792747 5200 log.cc:1196] T 7f0ea65032f64a6fb74b88fb4fc92f0b P e51a15732ba5427cbe7cf7466dbf9274: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/7f0ea65032f64a6fb74b88fb4fc92f0b
I20240502 07:59:48.793223 2923 catalog_manager.cc:4830] TS e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705): tablet 7f0ea65032f64a6fb74b88fb4fc92f0b (table rebalance_test_table_2 [id=242dd44a47a143d6ad62e53b54120b3a]) successfully deleted
I20240502 07:59:48.870218 3973 consensus_queue.cc:237] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:48.871661 3078 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 2 FOLLOWER]: Refusing update from remote peer 2a45c4829a5d41c4bc1c1e9c91277264: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 07:59:48.871666 3209 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 2 FOLLOWER]: Refusing update from remote peer 2a45c4829a5d41c4bc1c1e9c91277264: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 07:59:48.872119 5158 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:48.872365 5155 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:48.873970 5158 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:48.874526 3209 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:48.874516 3078 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:48.875612 2922 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:48.876577 2930 catalog_manager.cc:5484] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 reported cstate change: config changed from index 6 to 8, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New cstate: current_term: 2 leader_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:48.879128 3320 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 with delete_type TABLET_DATA_TOMBSTONED (TS e51a15732ba5427cbe7cf7466dbf9274 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:53650
I20240502 07:59:48.879442 5200 tablet_replica.cc:330] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274: stopping tablet replica
I20240502 07:59:48.879583 5200 raft_consensus.cc:2229] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:48.879711 5200 raft_consensus.cc:2258] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:48.880532 5200 ts_tablet_manager.cc:1875] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:48.882380 5200 ts_tablet_manager.cc:1888] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 07:59:48.882453 5200 log.cc:1196] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P e51a15732ba5427cbe7cf7466dbf9274: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/3d73ac6edc2549e7a4c7dbc2bb91d750
I20240502 07:59:48.882867 2923 catalog_manager.cc:4830] TS e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705): tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 (table rebalance_test_table_2 [id=242dd44a47a143d6ad62e53b54120b3a]) successfully deleted
I20240502 07:59:49.061534 3973 raft_consensus.cc:1203] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 07:59:49.177994 3209 consensus_queue.cc:237] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:49.179301 4104 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:49.179337 3341 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:49.179708 5217 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:49.179960 5048 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:49.181764 5216 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:49.183385 4104 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:49.183724 3341 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:49.184278 2933 catalog_manager.cc:5484] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:49.188591 3209 consensus_queue.cc:237] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:49.189869 4104 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:49.190074 3341 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:49.190291 5048 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:49.190457 5216 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 07:59:49.190569 3145 consensus_peers.cc:485] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c -> Peer 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979): Couldn't send request to peer 2a45c4829a5d41c4bc1c1e9c91277264. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3429dee9c9e24927992401f2af945375. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:49.192224 5216 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:49.192530 4104 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:49.194026 3341 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:49.194200 2923 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3429dee9c9e24927992401f2af945375 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:59:49.195480 2930 catalog_manager.cc:5484] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c reported cstate change: config changed from index 7 to 8, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:49.196619 3341 consensus_queue.cc:237] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:49.197826 4104 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:49.198194 5038 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:49.198246 3209 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:49.198714 5233 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:49.200659 5038 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:49.201056 3209 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:49.202028 4104 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:49.202950 2928 catalog_manager.cc:5484] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:49.206421 3341 consensus_queue.cc:237] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:49.207710 4104 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:49.208151 3209 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 07:59:49.208192 3277 consensus_peers.cc:485] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 -> Peer 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979): Couldn't send request to peer 2a45c4829a5d41c4bc1c1e9c91277264. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e10362e44ff490da447b508aedf4226. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:49.208590 5236 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:49.208775 5233 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:49.210377 5236 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:49.210839 3209 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:49.211968 2923 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0e10362e44ff490da447b508aedf4226 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:59:49.212342 4104 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } } }
I20240502 07:59:49.212781 2930 catalog_manager.cc:5484] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index 7 to 8, NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) added. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:49.622314 5217 raft_consensus.cc:1050] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c: attempting to promote NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 to VOTER
I20240502 07:59:49.622746 5217 consensus_queue.cc:237] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:49.624248 4104 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:49.624249 3341 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:49.624300 3973 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:49.624660 5217 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:49.624891 5268 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:49.625062 5216 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:49.627102 5268 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:49.627560 4104 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:49.627948 3341 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:49.629689 3973 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:49.629956 2930 catalog_manager.cc:5484] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c reported cstate change: config changed from index 8 to 9, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:49.633589 3209 consensus_queue.cc:237] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:49.634606 3973 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:49.634723 4104 raft_consensus.cc:1261] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:49.634969 5268 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:49.635183 5217 consensus_queue.cc:1035] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:49.636714 5216 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:49.637106 4104 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:49.637157 3973 raft_consensus.cc:2932] T e6b4e558aad64651aad857a4cbfac9d9 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:49.638206 2923 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6b4e558aad64651aad857a4cbfac9d9 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:49.639273 2929 catalog_manager.cc:5484] T e6b4e558aad64651aad857a4cbfac9d9 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: config changed from index 9 to 10, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:49.645696 3320 tablet_service.cc:1514] Processing DeleteTablet for tablet e6b4e558aad64651aad857a4cbfac9d9 with delete_type TABLET_DATA_TOMBSTONED (TS e51a15732ba5427cbe7cf7466dbf9274 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53650
I20240502 07:59:49.646358 5276 tablet_replica.cc:330] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274: stopping tablet replica
I20240502 07:59:49.646584 5276 raft_consensus.cc:2229] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:49.646826 5276 raft_consensus.cc:2258] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:49.647557 5276 ts_tablet_manager.cc:1875] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:49.649709 5276 ts_tablet_manager.cc:1888] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 07:59:49.649791 5276 log.cc:1196] T e6b4e558aad64651aad857a4cbfac9d9 P e51a15732ba5427cbe7cf7466dbf9274: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/e6b4e558aad64651aad857a4cbfac9d9
I20240502 07:59:49.650398 2923 catalog_manager.cc:4830] TS e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705): tablet e6b4e558aad64651aad857a4cbfac9d9 (table rebalance_test_table_2 [id=242dd44a47a143d6ad62e53b54120b3a]) successfully deleted
I20240502 07:59:49.766235 5288 ts_tablet_manager.cc:902] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264: Initiating tablet copy from peer e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705)
I20240502 07:59:49.766705 5288 tablet_copy_client.cc:308] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.131:43705
I20240502 07:59:49.767267 3361 tablet_copy_service.cc:140] P e51a15732ba5427cbe7cf7466dbf9274: Received BeginTabletCopySession request for tablet 0e10362e44ff490da447b508aedf4226 from peer 2a45c4829a5d41c4bc1c1e9c91277264 ({username='slave'} at 127.18.13.133:40463)
I20240502 07:59:49.767376 3361 tablet_copy_service.cc:161] P e51a15732ba5427cbe7cf7466dbf9274: Beginning new tablet copy session on tablet 0e10362e44ff490da447b508aedf4226 from peer 2a45c4829a5d41c4bc1c1e9c91277264 at {username='slave'} at 127.18.13.133:40463: session id = 2a45c4829a5d41c4bc1c1e9c91277264-0e10362e44ff490da447b508aedf4226
I20240502 07:59:49.768157 3361 tablet_copy_source_session.cc:205] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:49.769095 5293 ts_tablet_manager.cc:902] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264: Initiating tablet copy from peer 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:49.769131 5288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e10362e44ff490da447b508aedf4226. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:49.769627 5293 tablet_copy_client.cc:308] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:42851
I20240502 07:59:49.770145 3230 tablet_copy_service.cc:140] P 3dc087b5db164c89bea2988820a9b31c: Received BeginTabletCopySession request for tablet 3429dee9c9e24927992401f2af945375 from peer 2a45c4829a5d41c4bc1c1e9c91277264 ({username='slave'} at 127.18.13.133:53805)
I20240502 07:59:49.770251 3230 tablet_copy_service.cc:161] P 3dc087b5db164c89bea2988820a9b31c: Beginning new tablet copy session on tablet 3429dee9c9e24927992401f2af945375 from peer 2a45c4829a5d41c4bc1c1e9c91277264 at {username='slave'} at 127.18.13.133:53805: session id = 2a45c4829a5d41c4bc1c1e9c91277264-3429dee9c9e24927992401f2af945375
I20240502 07:59:49.770900 3230 tablet_copy_source_session.cc:205] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:49.771049 5288 tablet_copy_client.cc:783] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:49.771209 5288 tablet_copy_client.cc:647] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:49.771704 5293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3429dee9c9e24927992401f2af945375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:49.772552 5288 tablet_copy_client.cc:523] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:49.773681 5293 tablet_copy_client.cc:783] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:49.773764 5288 tablet_bootstrap.cc:492] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap starting.
I20240502 07:59:49.773821 5293 tablet_copy_client.cc:647] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:49.774995 5293 tablet_copy_client.cc:523] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:49.775964 5293 tablet_bootstrap.cc:492] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap starting.
I20240502 07:59:49.793182 5288 tablet_bootstrap.cc:492] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:49.793483 5288 tablet_bootstrap.cc:492] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap complete.
I20240502 07:59:49.793670 5288 ts_tablet_manager.cc:1367] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20240502 07:59:49.794103 5288 raft_consensus.cc:348] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:49.794215 5288 raft_consensus.cc:729] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a45c4829a5d41c4bc1c1e9c91277264, State: Initialized, Role: LEARNER
I20240502 07:59:49.794298 5288 consensus_queue.cc:260] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:49.794976 5288 ts_tablet_manager.cc:1398] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 07:59:49.795464 5293 tablet_bootstrap.cc:492] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:49.795608 3361 tablet_copy_service.cc:342] P e51a15732ba5427cbe7cf7466dbf9274: Request end of tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-0e10362e44ff490da447b508aedf4226 received from {username='slave'} at 127.18.13.133:40463
I20240502 07:59:49.795718 3361 tablet_copy_service.cc:433] P e51a15732ba5427cbe7cf7466dbf9274: ending tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-0e10362e44ff490da447b508aedf4226 on tablet 0e10362e44ff490da447b508aedf4226 with peer 2a45c4829a5d41c4bc1c1e9c91277264
I20240502 07:59:49.795837 5293 tablet_bootstrap.cc:492] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264: Bootstrap complete.
I20240502 07:59:49.795984 5293 ts_tablet_manager.cc:1367] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent bootstrapping tablet: real 0.020s user 0.020s sys 0.000s
I20240502 07:59:49.796414 5293 raft_consensus.cc:348] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:49.796522 5293 raft_consensus.cc:729] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2a45c4829a5d41c4bc1c1e9c91277264, State: Initialized, Role: LEARNER
I20240502 07:59:49.796726 5293 consensus_queue.cc:260] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: NON_VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: true } }
I20240502 07:59:49.797070 5293 ts_tablet_manager.cc:1398] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:49.797564 3230 tablet_copy_service.cc:342] P 3dc087b5db164c89bea2988820a9b31c: Request end of tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-3429dee9c9e24927992401f2af945375 received from {username='slave'} at 127.18.13.133:53805
I20240502 07:59:49.797641 3230 tablet_copy_service.cc:433] P 3dc087b5db164c89bea2988820a9b31c: ending tablet copy session 2a45c4829a5d41c4bc1c1e9c91277264-3429dee9c9e24927992401f2af945375 on tablet 3429dee9c9e24927992401f2af945375 with peer 2a45c4829a5d41c4bc1c1e9c91277264
I20240502 07:59:49.868261 3078 consensus_queue.cc:237] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:49.869551 3973 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:49.869552 3341 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:49.869923 5307 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:49.870090 4597 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:49.871959 4597 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:49.872407 3341 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:49.874415 2930 catalog_manager.cc:5484] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "d11d320786dd4f1094ce083e43bfd5b5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:49.875036 3973 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:49.878791 3078 consensus_queue.cc:237] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:49.880437 3973 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:49.880447 3341 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:49.881254 5307 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:49.882233 5310 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:49.884132 5307 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:49.885607 3341 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:49.886152 3973 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
W20240502 07:59:49.887392 3012 consensus_peers.cc:485] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 -> Peer 73bb095f4df448ae81da20d1a671547c (127.18.13.132:35129): Couldn't send request to peer 73bb095f4df448ae81da20d1a671547c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cdfb5d62286445b98e04d397fa8ffdb1. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:49.887388 2921 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cdfb5d62286445b98e04d397fa8ffdb1 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:59:49.887928 2929 catalog_manager.cc:5484] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index 7 to 8, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "d11d320786dd4f1094ce083e43bfd5b5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:50.170918 3973 raft_consensus.cc:1203] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 07:59:50.207114 3973 raft_consensus.cc:1203] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 07:59:50.297586 5348 ts_tablet_manager.cc:902] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c: Initiating tablet copy from peer d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063)
I20240502 07:59:50.298084 5348 tablet_copy_client.cc:308] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37063
I20240502 07:59:50.298609 3098 tablet_copy_service.cc:140] P d11d320786dd4f1094ce083e43bfd5b5: Received BeginTabletCopySession request for tablet cdfb5d62286445b98e04d397fa8ffdb1 from peer 73bb095f4df448ae81da20d1a671547c ({username='slave'} at 127.18.13.132:37725)
I20240502 07:59:50.298725 3098 tablet_copy_service.cc:161] P d11d320786dd4f1094ce083e43bfd5b5: Beginning new tablet copy session on tablet cdfb5d62286445b98e04d397fa8ffdb1 from peer 73bb095f4df448ae81da20d1a671547c at {username='slave'} at 127.18.13.132:37725: session id = 73bb095f4df448ae81da20d1a671547c-cdfb5d62286445b98e04d397fa8ffdb1
I20240502 07:59:50.299521 3098 tablet_copy_source_session.cc:205] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:50.300422 5348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfb5d62286445b98e04d397fa8ffdb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:50.302220 5348 tablet_copy_client.cc:783] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:50.302383 5348 tablet_copy_client.cc:647] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:50.303579 5348 tablet_copy_client.cc:523] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:50.304657 5348 tablet_bootstrap.cc:492] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c: Bootstrap starting.
I20240502 07:59:50.325752 5348 tablet_bootstrap.cc:492] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:50.326073 5348 tablet_bootstrap.cc:492] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c: Bootstrap complete.
I20240502 07:59:50.326283 5348 ts_tablet_manager.cc:1367] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.000s
I20240502 07:59:50.326740 5348 raft_consensus.cc:348] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:50.326853 5348 raft_consensus.cc:729] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 73bb095f4df448ae81da20d1a671547c, State: Initialized, Role: LEARNER
I20240502 07:59:50.326952 5348 consensus_queue.cc:260] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:50.327243 5348 ts_tablet_manager.cc:1398] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:50.327853 3098 tablet_copy_service.cc:342] P d11d320786dd4f1094ce083e43bfd5b5: Request end of tablet copy session 73bb095f4df448ae81da20d1a671547c-cdfb5d62286445b98e04d397fa8ffdb1 received from {username='slave'} at 127.18.13.132:37725
I20240502 07:59:50.327948 3098 tablet_copy_service.cc:433] P d11d320786dd4f1094ce083e43bfd5b5: ending tablet copy session 73bb095f4df448ae81da20d1a671547c-cdfb5d62286445b98e04d397fa8ffdb1 on tablet cdfb5d62286445b98e04d397fa8ffdb1 with peer 73bb095f4df448ae81da20d1a671547c
I20240502 07:59:50.588969 5345 raft_consensus.cc:1050] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c: attempting to promote NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 to VOTER
I20240502 07:59:50.589377 5345 consensus_queue.cc:237] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:50.590811 4104 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:50.590854 3973 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:50.590854 3341 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:50.591289 5344 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:50.591497 5217 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:50.591758 5345 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:50.593750 5344 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:50.594094 4104 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:50.595628 3973 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:50.595817 3341 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:50.597681 2928 catalog_manager.cc:5484] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c reported cstate change: config changed from index 8 to 9, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:50.601220 3209 consensus_queue.cc:237] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:50.602285 3973 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:50.602370 4104 raft_consensus.cc:1261] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:50.602672 5217 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:50.602847 5385 consensus_queue.cc:1035] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:50.604219 5385 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:50.604635 3973 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:50.604874 4104 raft_consensus.cc:2932] T 3429dee9c9e24927992401f2af945375 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:50.606061 2923 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3429dee9c9e24927992401f2af945375 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:50.606813 2933 catalog_manager.cc:5484] T 3429dee9c9e24927992401f2af945375 P 2a45c4829a5d41c4bc1c1e9c91277264 reported cstate change: config changed from index 9 to 10, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:50.609227 3320 tablet_service.cc:1514] Processing DeleteTablet for tablet 3429dee9c9e24927992401f2af945375 with delete_type TABLET_DATA_TOMBSTONED (TS e51a15732ba5427cbe7cf7466dbf9274 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53650
I20240502 07:59:50.609591 5392 tablet_replica.cc:330] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274: stopping tablet replica
I20240502 07:59:50.609735 5392 raft_consensus.cc:2229] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:50.609939 5392 raft_consensus.cc:2258] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:50.610862 5392 ts_tablet_manager.cc:1875] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:50.612614 5392 ts_tablet_manager.cc:1888] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 07:59:50.612679 5392 log.cc:1196] T 3429dee9c9e24927992401f2af945375 P e51a15732ba5427cbe7cf7466dbf9274: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/3429dee9c9e24927992401f2af945375
I20240502 07:59:50.613147 2923 catalog_manager.cc:4830] TS e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705): tablet 3429dee9c9e24927992401f2af945375 (table rebalance_test_table_1 [id=90a1f5d24e6f413eb4ef048ae0e4d46e]) successfully deleted
I20240502 07:59:50.670378 5347 raft_consensus.cc:1050] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274: attempting to promote NON_VOTER 2a45c4829a5d41c4bc1c1e9c91277264 to VOTER
I20240502 07:59:50.670794 5347 consensus_queue.cc:237] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:50.672293 3209 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:50.672293 4104 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:50.672402 3973 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 LEARNER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:50.672792 5038 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:50.673048 5394 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:50.673177 5346 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:50.675663 5346 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:50.676062 3973 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:50.677560 3209 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:50.678332 4104 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:50.678464 2930 catalog_manager.cc:5484] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264 reported cstate change: config changed from index 8 to 9, 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:50.769299 3341 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e10362e44ff490da447b508aedf4226"
dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59508
I20240502 07:59:50.769439 3341 raft_consensus.cc:595] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:50.852635 4104 raft_consensus.cc:1203] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 07:59:50.989329 3209 consensus_queue.cc:237] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } }
I20240502 07:59:50.990660 3341 raft_consensus.cc:1261] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:50.990780 3078 raft_consensus.cc:1261] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:59:50.991194 5345 consensus_queue.cc:1035] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:50.991470 5217 consensus_queue.cc:1035] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:59:50.993455 5217 raft_consensus.cc:2932] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } }
I20240502 07:59:50.994014 3341 raft_consensus.cc:2932] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } }
I20240502 07:59:50.996455 2929 catalog_manager.cc:5484] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } }
I20240502 07:59:50.996495 3076 raft_consensus.cc:2932] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } }
I20240502 07:59:51.001328 3209 consensus_queue.cc:237] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
W20240502 07:59:51.003082 3144 consensus_peers.cc:485] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c -> Peer 73bb095f4df448ae81da20d1a671547c (127.18.13.132:35129): Couldn't send request to peer 73bb095f4df448ae81da20d1a671547c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4ff93bab11949b0bdb739f5375e4d53. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:51.003082 3076 raft_consensus.cc:1261] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:51.003108 3341 raft_consensus.cc:1261] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:59:51.003535 5344 consensus_queue.cc:1035] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:51.003701 5345 consensus_queue.cc:1035] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:59:51.005437 5217 raft_consensus.cc:2932] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:51.007077 2923 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c4ff93bab11949b0bdb739f5375e4d53 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:59:51.007328 3341 raft_consensus.cc:2932] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:51.007519 3076 raft_consensus.cc:2932] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:51.010111 2928 catalog_manager.cc:5484] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index 4 to 5, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:51.026908 3341 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e10362e44ff490da447b508aedf4226"
dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59534
I20240502 07:59:51.027034 3341 raft_consensus.cc:595] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:51.077059 5394 raft_consensus.cc:979] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274: : Instructing follower 3dc087b5db164c89bea2988820a9b31c to start an election
I20240502 07:59:51.077188 5394 raft_consensus.cc:1067] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Signalling peer 3dc087b5db164c89bea2988820a9b31c to start an election
I20240502 07:59:51.077692 3209 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e10362e44ff490da447b508aedf4226"
dest_uuid: "3dc087b5db164c89bea2988820a9b31c"
from {username='slave'} at 127.18.13.131:47125
I20240502 07:59:51.077859 3209 raft_consensus.cc:482] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:51.077912 3209 raft_consensus.cc:3037] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:51.078794 3209 raft_consensus.cc:504] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:51.079262 3209 leader_election.cc:288] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 2 election: Requested vote from peers e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705), 73bb095f4df448ae81da20d1a671547c (127.18.13.132:35129), 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979)
I20240502 07:59:51.079676 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e10362e44ff490da447b508aedf4226" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "2a45c4829a5d41c4bc1c1e9c91277264"
I20240502 07:59:51.079676 3341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e10362e44ff490da447b508aedf4226" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
I20240502 07:59:51.079737 4104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e10362e44ff490da447b508aedf4226" candidate_uuid: "3dc087b5db164c89bea2988820a9b31c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "73bb095f4df448ae81da20d1a671547c"
I20240502 07:59:51.079841 3341 raft_consensus.cc:3032] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:51.079836 3973 raft_consensus.cc:3037] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:51.079890 3341 raft_consensus.cc:729] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Running, Role: LEADER
I20240502 07:59:51.079948 4104 raft_consensus.cc:3037] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:51.080071 3341 consensus_queue.cc:260] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:51.080278 3341 raft_consensus.cc:3037] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:51.080960 4104 raft_consensus.cc:2454] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 2.
I20240502 07:59:51.080960 3973 raft_consensus.cc:2454] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 2.
I20240502 07:59:51.081321 3341 raft_consensus.cc:2454] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dc087b5db164c89bea2988820a9b31c in term 2.
I20240502 07:59:51.081364 3144 leader_election.cc:302] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2a45c4829a5d41c4bc1c1e9c91277264, 3dc087b5db164c89bea2988820a9b31c, 73bb095f4df448ae81da20d1a671547c; no voters:
I20240502 07:59:51.081555 5344 raft_consensus.cc:2781] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:51.081662 5344 raft_consensus.cc:686] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 2 LEADER]: Becoming Leader. State: Replica: 3dc087b5db164c89bea2988820a9b31c, State: Running, Role: LEADER
I20240502 07:59:51.081808 5344 consensus_queue.cc:237] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:51.083557 2933 catalog_manager.cc:5484] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: term changed from 1 to 2, leader changed from e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) to 3dc087b5db164c89bea2988820a9b31c (127.18.13.130). New cstate: current_term: 2 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:51.329762 5310 raft_consensus.cc:1050] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5: attempting to promote NON_VOTER 73bb095f4df448ae81da20d1a671547c to VOTER
I20240502 07:59:51.330212 5310 consensus_queue.cc:237] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:51.331693 4104 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:51.331748 3341 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:51.331737 3973 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:59:51.332146 5464 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:51.332365 5463 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:51.332543 5310 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:51.334493 5464 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:51.334897 4104 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:51.334874 3973 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:51.336477 3341 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:51.337546 2929 catalog_manager.cc:5484] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 reported cstate change: config changed from index 8 to 9, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d11d320786dd4f1094ce083e43bfd5b5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:51.340641 3076 consensus_queue.cc:237] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:51.341897 3973 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:51.341989 4104 raft_consensus.cc:1261] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:59:51.342324 5310 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:51.342497 5463 consensus_queue.cc:1035] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:51.344180 5310 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:51.344650 4104 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:51.344756 3973 raft_consensus.cc:2932] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:51.345996 2921 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cdfb5d62286445b98e04d397fa8ffdb1 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:51.347198 2933 catalog_manager.cc:5484] T cdfb5d62286445b98e04d397fa8ffdb1 P 2a45c4829a5d41c4bc1c1e9c91277264 reported cstate change: config changed from index 9 to 10, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "d11d320786dd4f1094ce083e43bfd5b5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:51.349661 3320 tablet_service.cc:1514] Processing DeleteTablet for tablet cdfb5d62286445b98e04d397fa8ffdb1 with delete_type TABLET_DATA_TOMBSTONED (TS e51a15732ba5427cbe7cf7466dbf9274 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53650
I20240502 07:59:51.350033 5468 tablet_replica.cc:330] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274: stopping tablet replica
I20240502 07:59:51.350164 5468 raft_consensus.cc:2229] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:51.350348 5468 raft_consensus.cc:2258] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:51.351033 5468 ts_tablet_manager.cc:1875] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:51.352969 5468 ts_tablet_manager.cc:1888] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 07:59:51.353045 5468 log.cc:1196] T cdfb5d62286445b98e04d397fa8ffdb1 P e51a15732ba5427cbe7cf7466dbf9274: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/cdfb5d62286445b98e04d397fa8ffdb1
I20240502 07:59:51.353557 2923 catalog_manager.cc:4830] TS e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705): tablet cdfb5d62286445b98e04d397fa8ffdb1 (table rebalance_test_table_1 [id=90a1f5d24e6f413eb4ef048ae0e4d46e]) successfully deleted
I20240502 07:59:51.506943 4104 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 2 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:59:51.507474 5385 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:51.508055 3973 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:59:51.508456 5385 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:51.511234 3341 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:59:51.511637 5385 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:51.513751 3209 consensus_queue.cc:237] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:51.514896 4104 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 2 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 07:59:51.514994 3973 raft_consensus.cc:1261] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 07:59:51.515380 5485 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:51.515564 5385 consensus_queue.cc:1035] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:51.517100 5385 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:51.517834 5490 ts_tablet_manager.cc:902] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c: Initiating tablet copy from peer 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851)
I20240502 07:59:51.517763 3973 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:51.518429 5490 tablet_copy_client.cc:308] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.130:42851
I20240502 07:59:51.518926 2923 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e10362e44ff490da447b508aedf4226 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:51.519265 3230 tablet_copy_service.cc:140] P 3dc087b5db164c89bea2988820a9b31c: Received BeginTabletCopySession request for tablet c4ff93bab11949b0bdb739f5375e4d53 from peer 73bb095f4df448ae81da20d1a671547c ({username='slave'} at 127.18.13.132:34233)
I20240502 07:59:51.519378 3230 tablet_copy_service.cc:161] P 3dc087b5db164c89bea2988820a9b31c: Beginning new tablet copy session on tablet c4ff93bab11949b0bdb739f5375e4d53 from peer 73bb095f4df448ae81da20d1a671547c at {username='slave'} at 127.18.13.132:34233: session id = 73bb095f4df448ae81da20d1a671547c-c4ff93bab11949b0bdb739f5375e4d53
I20240502 07:59:51.520233 3230 tablet_copy_source_session.cc:205] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:51.520592 4104 raft_consensus.cc:2932] T 0e10362e44ff490da447b508aedf4226 P 73bb095f4df448ae81da20d1a671547c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:51.520910 2928 catalog_manager.cc:5484] T 0e10362e44ff490da447b508aedf4226 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: config changed from index 9 to 11, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New cstate: current_term: 2 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:51.521404 5490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ff93bab11949b0bdb739f5375e4d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.523634 5490 tablet_copy_client.cc:783] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:51.523816 5490 tablet_copy_client.cc:647] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:51.523918 3320 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e10362e44ff490da447b508aedf4226 with delete_type TABLET_DATA_TOMBSTONED (TS e51a15732ba5427cbe7cf7466dbf9274 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:53650
I20240502 07:59:51.524253 5468 tablet_replica.cc:330] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274: stopping tablet replica
I20240502 07:59:51.524391 5468 raft_consensus.cc:2229] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:51.524505 5468 raft_consensus.cc:2258] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:51.525312 5468 ts_tablet_manager.cc:1875] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:51.526722 5490 tablet_copy_client.cc:523] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:51.527351 5468 ts_tablet_manager.cc:1888] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 07:59:51.527415 5468 log.cc:1196] T 0e10362e44ff490da447b508aedf4226 P e51a15732ba5427cbe7cf7466dbf9274: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/0e10362e44ff490da447b508aedf4226
I20240502 07:59:51.527928 5490 tablet_bootstrap.cc:492] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c: Bootstrap starting.
I20240502 07:59:51.527907 2923 catalog_manager.cc:4830] TS e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705): tablet 0e10362e44ff490da447b508aedf4226 (table rebalance_test_table_0 [id=38b61401424d44728fb0d5830c0766fb]) successfully deleted
I20240502 07:59:51.547166 5490 tablet_bootstrap.cc:492] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:51.547479 5490 tablet_bootstrap.cc:492] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c: Bootstrap complete.
I20240502 07:59:51.547672 5490 ts_tablet_manager.cc:1367] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.001s
I20240502 07:59:51.548146 5490 raft_consensus.cc:348] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:51.548264 5490 raft_consensus.cc:729] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 73bb095f4df448ae81da20d1a671547c, State: Initialized, Role: LEARNER
I20240502 07:59:51.548349 5490 consensus_queue.cc:260] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:51.548763 5490 ts_tablet_manager.cc:1398] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240502 07:59:51.549548 3230 tablet_copy_service.cc:342] P 3dc087b5db164c89bea2988820a9b31c: Request end of tablet copy session 73bb095f4df448ae81da20d1a671547c-c4ff93bab11949b0bdb739f5375e4d53 received from {username='slave'} at 127.18.13.132:34233
I20240502 07:59:51.549635 3230 tablet_copy_service.cc:433] P 3dc087b5db164c89bea2988820a9b31c: ending tablet copy session 73bb095f4df448ae81da20d1a671547c-c4ff93bab11949b0bdb739f5375e4d53 on tablet c4ff93bab11949b0bdb739f5375e4d53 with peer 73bb095f4df448ae81da20d1a671547c
I20240502 07:59:51.672490 3341 consensus_queue.cc:237] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } }
I20240502 07:59:51.673792 3973 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:51.673822 3076 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:51.674242 5394 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:51.674597 5347 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:51.676172 5394 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:51.676604 3973 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:51.676930 3076 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } }
I20240502 07:59:51.679193 2930 catalog_manager.cc:5484] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:51.682540 3341 consensus_queue.cc:237] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: true } }
W20240502 07:59:51.684237 3278 consensus_peers.cc:485] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 -> Peer 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851): Couldn't send request to peer 3dc087b5db164c89bea2988820a9b31c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:51.684240 3076 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:51.684651 5394 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:51.684784 3973 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Refusing update from remote peer e51a15732ba5427cbe7cf7466dbf9274: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:59:51.685185 5394 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:59:51.687420 5347 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: true } } }
I20240502 07:59:51.688119 3076 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: true } } }
I20240502 07:59:51.688205 3973 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: true } } }
I20240502 07:59:51.690554 2928 catalog_manager.cc:5484] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 reported cstate change: config changed from index 7 to 8, NON_VOTER 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) added. New cstate: current_term: 1 leader_uuid: "e51a15732ba5427cbe7cf7466dbf9274" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:51.691079 2923 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8229b8193e454089b8e7e72973fe5e65 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:59:51.695906 3341 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8229b8193e454089b8e7e72973fe5e65"
dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59564
I20240502 07:59:51.696059 3341 raft_consensus.cc:595] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:52.025168 4104 raft_consensus.cc:1203] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:59:52.209436 5549 raft_consensus.cc:979] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274: : Instructing follower d11d320786dd4f1094ce083e43bfd5b5 to start an election
I20240502 07:59:52.209585 5549 raft_consensus.cc:1067] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Signalling peer d11d320786dd4f1094ce083e43bfd5b5 to start an election
I20240502 07:59:52.210191 3076 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8229b8193e454089b8e7e72973fe5e65"
dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5"
from {username='slave'} at 127.18.13.131:36723
I20240502 07:59:52.210374 3076 raft_consensus.cc:482] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:52.210439 3076 raft_consensus.cc:3037] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:52.211293 3076 raft_consensus.cc:504] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: true } }
I20240502 07:59:52.211692 3076 leader_election.cc:288] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [CANDIDATE]: Term 2 election: Requested vote from peers e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705), 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979)
I20240502 07:59:52.212101 3341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229b8193e454089b8e7e72973fe5e65" candidate_uuid: "d11d320786dd4f1094ce083e43bfd5b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e51a15732ba5427cbe7cf7466dbf9274"
I20240502 07:59:52.212237 3341 raft_consensus.cc:3032] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:52.212291 3341 raft_consensus.cc:729] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e51a15732ba5427cbe7cf7466dbf9274, State: Running, Role: LEADER
I20240502 07:59:52.212261 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8229b8193e454089b8e7e72973fe5e65" candidate_uuid: "d11d320786dd4f1094ce083e43bfd5b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "2a45c4829a5d41c4bc1c1e9c91277264"
I20240502 07:59:52.212390 3973 raft_consensus.cc:3037] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:52.212460 3341 consensus_queue.cc:260] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: true } }
I20240502 07:59:52.212687 3341 raft_consensus.cc:3037] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:52.213452 3973 raft_consensus.cc:2454] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d11d320786dd4f1094ce083e43bfd5b5 in term 2.
I20240502 07:59:52.213677 3341 raft_consensus.cc:2454] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d11d320786dd4f1094ce083e43bfd5b5 in term 2.
I20240502 07:59:52.213871 3013 leader_election.cc:302] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a45c4829a5d41c4bc1c1e9c91277264, d11d320786dd4f1094ce083e43bfd5b5; no voters:
I20240502 07:59:52.214143 5463 raft_consensus.cc:2781] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:52.214340 5463 raft_consensus.cc:686] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 2 LEADER]: Becoming Leader. State: Replica: d11d320786dd4f1094ce083e43bfd5b5, State: Running, Role: LEADER
I20240502 07:59:52.214509 5463 consensus_queue.cc:237] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: true } }
I20240502 07:59:52.216262 2930 catalog_manager.cc:5484] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 reported cstate change: term changed from 1 to 2, leader changed from e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) to d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129). New cstate: current_term: 2 leader_uuid: "d11d320786dd4f1094ce083e43bfd5b5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:52.612922 3973 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 07:59:52.613471 5463 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:52.614426 3341 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 07:59:52.614846 5551 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 07:59:52.617115 3014 consensus_peers.cc:485] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 -> Peer 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851): Couldn't send request to peer 3dc087b5db164c89bea2988820a9b31c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:52.642540 5580 raft_consensus.cc:1050] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c: attempting to promote NON_VOTER 73bb095f4df448ae81da20d1a671547c to VOTER
I20240502 07:59:52.643002 5580 consensus_queue.cc:237] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:52.644495 4104 raft_consensus.cc:1261] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c [term 1 LEARNER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:52.644594 3341 raft_consensus.cc:1261] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:52.644969 5580 consensus_queue.cc:1035] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:52.645272 3076 raft_consensus.cc:1261] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:59:52.645243 5485 consensus_queue.cc:1035] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:52.645642 5485 consensus_queue.cc:1035] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:59:52.647740 5585 raft_consensus.cc:2932] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:52.648126 3076 raft_consensus.cc:2932] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:52.648262 4104 raft_consensus.cc:2932] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:52.650507 3341 raft_consensus.cc:2932] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:52.650674 2933 catalog_manager.cc:5484] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 reported cstate change: config changed from index 5 to 6, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:52.654120 3209 consensus_queue.cc:237] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:52.655210 4104 raft_consensus.cc:1261] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:52.655251 3076 raft_consensus.cc:1261] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Refusing update from remote peer 3dc087b5db164c89bea2988820a9b31c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:59:52.655656 5485 consensus_queue.cc:1035] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:52.655851 5580 consensus_queue.cc:1035] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:59:52.657392 5580 raft_consensus.cc:2932] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:52.657841 4104 raft_consensus.cc:2932] T c4ff93bab11949b0bdb739f5375e4d53 P 73bb095f4df448ae81da20d1a671547c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:52.657841 3076 raft_consensus.cc:2932] T c4ff93bab11949b0bdb739f5375e4d53 P d11d320786dd4f1094ce083e43bfd5b5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:52.658782 2923 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c4ff93bab11949b0bdb739f5375e4d53 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:52.659597 2929 catalog_manager.cc:5484] T c4ff93bab11949b0bdb739f5375e4d53 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: config changed from index 6 to 7, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New cstate: current_term: 1 leader_uuid: "3dc087b5db164c89bea2988820a9b31c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:52.662006 3320 tablet_service.cc:1514] Processing DeleteTablet for tablet c4ff93bab11949b0bdb739f5375e4d53 with delete_type TABLET_DATA_TOMBSTONED (TS e51a15732ba5427cbe7cf7466dbf9274 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53650
I20240502 07:59:52.662493 5586 tablet_replica.cc:330] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274: stopping tablet replica
I20240502 07:59:52.662683 5586 raft_consensus.cc:2229] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:52.662892 5586 raft_consensus.cc:2258] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:52.663555 5586 ts_tablet_manager.cc:1875] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:52.665484 5586 ts_tablet_manager.cc:1888] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 07:59:52.665557 5586 log.cc:1196] T c4ff93bab11949b0bdb739f5375e4d53 P e51a15732ba5427cbe7cf7466dbf9274: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/c4ff93bab11949b0bdb739f5375e4d53
I20240502 07:59:52.666111 2923 catalog_manager.cc:4830] TS e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705): tablet c4ff93bab11949b0bdb739f5375e4d53 (table rebalance_test_table_0 [id=38b61401424d44728fb0d5830c0766fb]) successfully deleted
I20240502 07:59:53.015311 5602 ts_tablet_manager.cc:902] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: Initiating tablet copy from peer d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063)
I20240502 07:59:53.016151 5602 tablet_copy_client.cc:308] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.129:37063
I20240502 07:59:53.016785 3098 tablet_copy_service.cc:140] P d11d320786dd4f1094ce083e43bfd5b5: Received BeginTabletCopySession request for tablet 8229b8193e454089b8e7e72973fe5e65 from peer 3dc087b5db164c89bea2988820a9b31c ({username='slave'} at 127.18.13.130:42835)
I20240502 07:59:53.016893 3098 tablet_copy_service.cc:161] P d11d320786dd4f1094ce083e43bfd5b5: Beginning new tablet copy session on tablet 8229b8193e454089b8e7e72973fe5e65 from peer 3dc087b5db164c89bea2988820a9b31c at {username='slave'} at 127.18.13.130:42835: session id = 3dc087b5db164c89bea2988820a9b31c-8229b8193e454089b8e7e72973fe5e65
I20240502 07:59:53.017652 3098 tablet_copy_source_session.cc:205] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:53.018692 5602 ts_tablet_manager.cc:1875] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 07:59:53.020774 5602 ts_tablet_manager.cc:1888] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 07:59:53.020944 5602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8229b8193e454089b8e7e72973fe5e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:53.021916 5602 tablet_copy_client.cc:783] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:53.022071 5602 tablet_copy_client.cc:647] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:53.023437 5602 tablet_copy_client.cc:523] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:53.024727 5602 tablet_bootstrap.cc:492] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: Bootstrap starting.
I20240502 07:59:53.045172 5602 tablet_bootstrap.cc:492] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:53.045562 5602 tablet_bootstrap.cc:492] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: Bootstrap complete.
I20240502 07:59:53.045737 5602 ts_tablet_manager.cc:1367] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: Time spent bootstrapping tablet: real 0.021s user 0.020s sys 0.001s
I20240502 07:59:53.046137 5602 raft_consensus.cc:348] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: true } }
I20240502 07:59:53.046248 5602 raft_consensus.cc:729] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3dc087b5db164c89bea2988820a9b31c, State: Initialized, Role: LEARNER
I20240502 07:59:53.046332 5602 consensus_queue.cc:260] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: NON_VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: true } }
I20240502 07:59:53.046590 5602 ts_tablet_manager.cc:1398] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:53.047344 3098 tablet_copy_service.cc:342] P d11d320786dd4f1094ce083e43bfd5b5: Request end of tablet copy session 3dc087b5db164c89bea2988820a9b31c-8229b8193e454089b8e7e72973fe5e65 received from {username='slave'} at 127.18.13.130:42835
I20240502 07:59:53.047428 3098 tablet_copy_service.cc:433] P d11d320786dd4f1094ce083e43bfd5b5: ending tablet copy session 3dc087b5db164c89bea2988820a9b31c-8229b8193e454089b8e7e72973fe5e65 on tablet 8229b8193e454089b8e7e72973fe5e65 with peer 3dc087b5db164c89bea2988820a9b31c
I20240502 07:59:53.402657 3209 raft_consensus.cc:1203] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 2 LEARNER]: Deduplicated request from leader. Original: 1.8->[2.9-2.9] Dedup: 2.9->[]
I20240502 07:59:53.964298 5619 raft_consensus.cc:1050] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5: attempting to promote NON_VOTER 3dc087b5db164c89bea2988820a9b31c to VOTER
I20240502 07:59:53.964687 5619 consensus_queue.cc:237] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: false } }
I20240502 07:59:53.965993 3209 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 2 LEARNER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:59:53.966080 3341 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:59:53.966147 3973 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:59:53.966475 5633 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:53.966727 5632 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:53.966903 5619 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:53.968950 5633 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: false } } }
I20240502 07:59:53.969509 3209 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: false } } }
I20240502 07:59:53.970168 3341 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: false } } }
I20240502 07:59:53.971246 3973 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: false } } }
I20240502 07:59:53.973529 2928 catalog_manager.cc:5484] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 reported cstate change: config changed from index 8 to 10, 3dc087b5db164c89bea2988820a9b31c (127.18.13.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d11d320786dd4f1094ce083e43bfd5b5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51a15732ba5427cbe7cf7466dbf9274" member_type: VOTER last_known_addr { host: "127.18.13.131" port: 43705 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:53.976722 3076 consensus_queue.cc:237] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: false } }
I20240502 07:59:53.977900 3209 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 2 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 07:59:53.978332 5619 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:53.978529 3973 raft_consensus.cc:1261] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Refusing update from remote peer d11d320786dd4f1094ce083e43bfd5b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 07:59:53.979107 5619 consensus_queue.cc:1035] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:53.979938 5630 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P d11d320786dd4f1094ce083e43bfd5b5 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: false } } }
I20240502 07:59:53.980327 3209 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: false } } }
I20240502 07:59:53.981791 2921 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8229b8193e454089b8e7e72973fe5e65 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:53.982501 2930 catalog_manager.cc:5484] T 8229b8193e454089b8e7e72973fe5e65 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: config changed from index 10 to 11, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New cstate: current_term: 2 leader_uuid: "d11d320786dd4f1094ce083e43bfd5b5" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: false } } }
I20240502 07:59:53.983211 3973 raft_consensus.cc:2932] T 8229b8193e454089b8e7e72973fe5e65 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } attrs { promote: false } } }
I20240502 07:59:53.985033 3320 tablet_service.cc:1514] Processing DeleteTablet for tablet 8229b8193e454089b8e7e72973fe5e65 with delete_type TABLET_DATA_TOMBSTONED (TS e51a15732ba5427cbe7cf7466dbf9274 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:53650
I20240502 07:59:53.985419 5643 tablet_replica.cc:330] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274: stopping tablet replica
I20240502 07:59:53.985584 5643 raft_consensus.cc:2229] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:53.985781 5643 raft_consensus.cc:2258] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:53.986531 5643 ts_tablet_manager.cc:1875] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:53.988638 5643 ts_tablet_manager.cc:1888] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 07:59:53.988713 5643 log.cc:1196] T 8229b8193e454089b8e7e72973fe5e65 P e51a15732ba5427cbe7cf7466dbf9274: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-2/wal/wals/8229b8193e454089b8e7e72973fe5e65
I20240502 07:59:53.989246 2923 catalog_manager.cc:4830] TS e51a15732ba5427cbe7cf7466dbf9274 (127.18.13.131:43705): tablet 8229b8193e454089b8e7e72973fe5e65 (table rebalance_test_table_0 [id=38b61401424d44728fb0d5830c0766fb]) successfully deleted
I20240502 07:59:54.265946 3188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:54.268438 3057 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:54.268755 4085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:54.268880 3954 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:54.276471 3320 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:54.367712 3954 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:54.368377 3188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:54.369479 4085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:54.376773 3320 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:54.390204 3057 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:54.475936 3188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:54.478689 4085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:54.478791 3057 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:54.479658 3954 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:54.493836 3320 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:54.583755 3954 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:54.584146 3057 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:54.584398 3320 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:54.584663 3188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:54.584832 4085 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:54.626459 3973 consensus_queue.cc:237] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:54.627995 3209 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 2 FOLLOWER]: Refusing update from remote peer 2a45c4829a5d41c4bc1c1e9c91277264: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 07:59:54.628360 3076 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 2 FOLLOWER]: Refusing update from remote peer 2a45c4829a5d41c4bc1c1e9c91277264: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 07:59:54.628480 5792 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:54.628871 5792 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:59:54.630343 5791 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:54.630741 3209 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
W20240502 07:59:54.632954 3908 consensus_peers.cc:485] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 -> Peer 73bb095f4df448ae81da20d1a671547c (127.18.13.132:35129): Couldn't send request to peer 73bb095f4df448ae81da20d1a671547c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d73ac6edc2549e7a4c7dbc2bb91d750. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:54.633157 2930 catalog_manager.cc:5484] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 reported cstate change: config changed from index 8 to 9, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New cstate: current_term: 2 leader_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:54.633877 3076 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 73bb095f4df448ae81da20d1a671547c (127.18.13.132) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } } }
I20240502 07:59:55.123136 5814 ts_tablet_manager.cc:902] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c: Initiating tablet copy from peer 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979)
I20240502 07:59:55.123612 5814 tablet_copy_client.cc:308] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Beginning tablet copy session from remote peer at address 127.18.13.133:40979
I20240502 07:59:55.124280 3994 tablet_copy_service.cc:140] P 2a45c4829a5d41c4bc1c1e9c91277264: Received BeginTabletCopySession request for tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 from peer 73bb095f4df448ae81da20d1a671547c ({username='slave'} at 127.18.13.132:36131)
I20240502 07:59:55.124440 3994 tablet_copy_service.cc:161] P 2a45c4829a5d41c4bc1c1e9c91277264: Beginning new tablet copy session on tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 from peer 73bb095f4df448ae81da20d1a671547c at {username='slave'} at 127.18.13.132:36131: session id = 73bb095f4df448ae81da20d1a671547c-3d73ac6edc2549e7a4c7dbc2bb91d750
I20240502 07:59:55.125577 3994 tablet_copy_source_session.cc:205] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:55.126678 5814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d73ac6edc2549e7a4c7dbc2bb91d750. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.128535 5814 tablet_copy_client.cc:783] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:55.128681 5814 tablet_copy_client.cc:647] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:55.130049 5814 tablet_copy_client.cc:523] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:55.131170 5814 tablet_bootstrap.cc:492] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c: Bootstrap starting.
I20240502 07:59:55.149833 5814 tablet_bootstrap.cc:492] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:55.150151 5814 tablet_bootstrap.cc:492] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c: Bootstrap complete.
I20240502 07:59:55.150338 5814 ts_tablet_manager.cc:1367] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.003s
I20240502 07:59:55.150810 5814 raft_consensus.cc:348] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:55.150935 5814 raft_consensus.cc:729] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 73bb095f4df448ae81da20d1a671547c, State: Initialized, Role: LEARNER
I20240502 07:59:55.151041 5814 consensus_queue.cc:260] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: NON_VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: true } }
I20240502 07:59:55.151319 5814 ts_tablet_manager.cc:1398] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:55.152065 3994 tablet_copy_service.cc:342] P 2a45c4829a5d41c4bc1c1e9c91277264: Request end of tablet copy session 73bb095f4df448ae81da20d1a671547c-3d73ac6edc2549e7a4c7dbc2bb91d750 received from {username='slave'} at 127.18.13.132:36131
I20240502 07:59:55.152179 3994 tablet_copy_service.cc:433] P 2a45c4829a5d41c4bc1c1e9c91277264: ending tablet copy session 73bb095f4df448ae81da20d1a671547c-3d73ac6edc2549e7a4c7dbc2bb91d750 on tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 with peer 73bb095f4df448ae81da20d1a671547c
I20240502 07:59:55.716522 4104 raft_consensus.cc:1203] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240502 07:59:56.109442 5835 raft_consensus.cc:1050] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: attempting to promote NON_VOTER 73bb095f4df448ae81da20d1a671547c to VOTER
I20240502 07:59:56.109982 5835 consensus_queue.cc:237] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:56.111513 3209 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 2 FOLLOWER]: Refusing update from remote peer 2a45c4829a5d41c4bc1c1e9c91277264: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:59:56.111521 4104 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [term 2 LEARNER]: Refusing update from remote peer 2a45c4829a5d41c4bc1c1e9c91277264: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:59:56.111675 3076 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 2 FOLLOWER]: Refusing update from remote peer 2a45c4829a5d41c4bc1c1e9c91277264: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:59:56.112072 5848 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:56.112306 5835 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:56.112478 5792 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:59:56.114333 5792 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:56.114758 4104 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:56.114833 3076 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:56.117241 2929 catalog_manager.cc:5484] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 reported cstate change: config changed from index 9 to 10, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:56.117231 3209 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 73bb095f4df448ae81da20d1a671547c (127.18.13.132) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:56.304196 3972 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d73ac6edc2549e7a4c7dbc2bb91d750"
dest_uuid: "2a45c4829a5d41c4bc1c1e9c91277264"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47468
I20240502 07:59:56.304364 3972 raft_consensus.cc:595] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 LEADER]: Received request to transfer leadership
I20240502 07:59:56.516369 3972 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d73ac6edc2549e7a4c7dbc2bb91d750"
dest_uuid: "2a45c4829a5d41c4bc1c1e9c91277264"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47486
I20240502 07:59:56.516525 3972 raft_consensus.cc:595] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 LEADER]: Received request to transfer leadership
I20240502 07:59:56.586408 5835 raft_consensus.cc:979] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: : Instructing follower 73bb095f4df448ae81da20d1a671547c to start an election
I20240502 07:59:56.586529 5835 raft_consensus.cc:1067] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 LEADER]: Signalling peer 73bb095f4df448ae81da20d1a671547c to start an election
I20240502 07:59:56.587144 4104 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d73ac6edc2549e7a4c7dbc2bb91d750"
dest_uuid: "73bb095f4df448ae81da20d1a671547c"
from {username='slave'} at 127.18.13.133:47293
I20240502 07:59:56.587285 4104 raft_consensus.cc:482] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:56.587333 4104 raft_consensus.cc:3037] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:56.588224 4104 raft_consensus.cc:504] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:56.588704 4104 leader_election.cc:288] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [CANDIDATE]: Term 3 election: Requested vote from peers d11d320786dd4f1094ce083e43bfd5b5 (127.18.13.129:37063), 3dc087b5db164c89bea2988820a9b31c (127.18.13.130:42851), 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979)
I20240502 07:59:56.589094 3209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d73ac6edc2549e7a4c7dbc2bb91d750" candidate_uuid: "73bb095f4df448ae81da20d1a671547c" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "3dc087b5db164c89bea2988820a9b31c"
I20240502 07:59:56.589114 3076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d73ac6edc2549e7a4c7dbc2bb91d750" candidate_uuid: "73bb095f4df448ae81da20d1a671547c" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "d11d320786dd4f1094ce083e43bfd5b5"
I20240502 07:59:56.589223 3209 raft_consensus.cc:3037] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:56.589270 3076 raft_consensus.cc:3037] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:56.589309 3972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d73ac6edc2549e7a4c7dbc2bb91d750" candidate_uuid: "73bb095f4df448ae81da20d1a671547c" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "2a45c4829a5d41c4bc1c1e9c91277264"
I20240502 07:59:56.589447 3972 raft_consensus.cc:3032] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 07:59:56.589491 3972 raft_consensus.cc:729] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2a45c4829a5d41c4bc1c1e9c91277264, State: Running, Role: LEADER
I20240502 07:59:56.589653 3972 consensus_queue.cc:260] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:56.589843 3972 raft_consensus.cc:3037] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:56.590080 3209 raft_consensus.cc:2454] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73bb095f4df448ae81da20d1a671547c in term 3.
I20240502 07:59:56.590304 3076 raft_consensus.cc:2454] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73bb095f4df448ae81da20d1a671547c in term 3.
I20240502 07:59:56.590658 4039 leader_election.cc:302] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3dc087b5db164c89bea2988820a9b31c, 73bb095f4df448ae81da20d1a671547c, d11d320786dd4f1094ce083e43bfd5b5; no voters:
I20240502 07:59:56.590781 3972 raft_consensus.cc:2454] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73bb095f4df448ae81da20d1a671547c in term 3.
I20240502 07:59:56.590847 5496 raft_consensus.cc:2781] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [term 3 FOLLOWER]: Leader election won for term 3
I20240502 07:59:56.591012 5496 raft_consensus.cc:686] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [term 3 LEADER]: Becoming Leader. State: Replica: 73bb095f4df448ae81da20d1a671547c, State: Running, Role: LEADER
I20240502 07:59:56.591184 5496 consensus_queue.cc:237] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:56.593008 2930 catalog_manager.cc:5484] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c reported cstate change: term changed from 2 to 3, leader changed from 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) to 73bb095f4df448ae81da20d1a671547c (127.18.13.132). New cstate: current_term: 3 leader_uuid: "73bb095f4df448ae81da20d1a671547c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:56.973857 3972 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 3 FOLLOWER]: Refusing update from remote peer 73bb095f4df448ae81da20d1a671547c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 07:59:56.974334 5496 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a45c4829a5d41c4bc1c1e9c91277264" member_type: VOTER last_known_addr { host: "127.18.13.133" port: 40979 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:57.123495 3209 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 3 FOLLOWER]: Refusing update from remote peer 73bb095f4df448ae81da20d1a671547c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 07:59:57.124073 5496 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:57.127358 3076 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 3 FOLLOWER]: Refusing update from remote peer 73bb095f4df448ae81da20d1a671547c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 07:59:57.128554 5892 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:59:57.632061 4105 consensus_queue.cc:237] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } }
I20240502 07:59:57.633139 3076 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 3 FOLLOWER]: Refusing update from remote peer 73bb095f4df448ae81da20d1a671547c: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 07:59:57.633167 3209 raft_consensus.cc:1261] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 3 FOLLOWER]: Refusing update from remote peer 73bb095f4df448ae81da20d1a671547c: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 07:59:57.633631 5912 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:59:57.633850 5913 consensus_queue.cc:1035] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:59:57.635319 5892 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 73bb095f4df448ae81da20d1a671547c [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:57.635880 3076 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P d11d320786dd4f1094ce083e43bfd5b5 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:57.635950 3209 raft_consensus.cc:2932] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:57.637147 2921 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:57.638389 2933 catalog_manager.cc:5484] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 3dc087b5db164c89bea2988820a9b31c reported cstate change: config changed from index 10 to 12, VOTER 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133) evicted. New cstate: current_term: 3 leader_uuid: "73bb095f4df448ae81da20d1a671547c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "d11d320786dd4f1094ce083e43bfd5b5" member_type: VOTER last_known_addr { host: "127.18.13.129" port: 37063 } } peers { permanent_uuid: "3dc087b5db164c89bea2988820a9b31c" member_type: VOTER last_known_addr { host: "127.18.13.130" port: 42851 } } peers { permanent_uuid: "73bb095f4df448ae81da20d1a671547c" member_type: VOTER last_known_addr { host: "127.18.13.132" port: 35129 } attrs { promote: false } } }
I20240502 07:59:57.641600 3954 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 with delete_type TABLET_DATA_TOMBSTONED (TS 2a45c4829a5d41c4bc1c1e9c91277264 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:54050
I20240502 07:59:57.642122 5918 tablet_replica.cc:330] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: stopping tablet replica
I20240502 07:59:57.642449 5918 raft_consensus.cc:2229] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:57.642658 5918 raft_consensus.cc:2258] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:57.648690 5918 ts_tablet_manager.cc:1875] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:57.650774 5918 ts_tablet_manager.cc:1888] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240502 07:59:57.650858 5918 log.cc:1196] T 3d73ac6edc2549e7a4c7dbc2bb91d750 P 2a45c4829a5d41c4bc1c1e9c91277264: Deleting WAL directory at /tmp/dist-test-taskeLuhys/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714636688619157-18486-0/raft_consensus-itest-cluster/ts-4/wal/wals/3d73ac6edc2549e7a4c7dbc2bb91d750
I20240502 07:59:57.651398 2922 catalog_manager.cc:4830] TS 2a45c4829a5d41c4bc1c1e9c91277264 (127.18.13.133:40979): tablet 3d73ac6edc2549e7a4c7dbc2bb91d750 (table rebalance_test_table_2 [id=242dd44a47a143d6ad62e53b54120b3a]) successfully deleted
I20240502 07:59:57.810740 4085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:57.810981 3954 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:57.812294 3188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:57.820282 3057 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:57.821470 3320 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:57.901417 3954 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:57.904973 3188 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:57.905063 3057 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:57.905889 4085 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:57.926852 3320 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:57.957588 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 2996
I20240502 07:59:57.964475 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 3127
I20240502 07:59:57.971429 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 3259
I20240502 07:59:57.978435 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 4023
I20240502 07:59:57.984877 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 3892
I20240502 07:59:57.991178 18486 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskeLuhys/build/debug/bin/kudu with pid 2905
2024-05-02T07:59:57Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2 (22533 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (36673 ms total)
[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (109391 ms total)
[ PASSED ] 6 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00>
1 FAILED TEST